Total runs:
37.6K
Run Growth:
-15.7K
Growth Rate:
-41.69%
Sponsored by
Vmake Labs Video Enhancer
jukofyork / miquplus-midnight-70b
Model's Last Updated: June 04 2024
text-generation
Introduction of miquplus-midnight-70b
Model Details of miquplus-midnight-70b
A creative writing model with 32k context, created by adding the essence of "what makes Midnight-Rose-70B-v2.0.3 different to Llama-2-70b-hf " onto miqu-1-70b .
Prompting format
Vicuna format is preferred:
USER: {prompt} ASSISTANT:
Mistral and Alpaca formats are also supported:
[INST] {prompt} [/INST]
### Instruction:
{prompt}
### Response:
Licence and usage restrictions
miqu-1-70b-sf is a dequantized version of the miqu-1-70b model leaked from MistralAI. All miqu-derived models, including this merge, are suitable for non-commercial, personal use only.
Mergekit configuration
The following YAML configuration was used to produce this model:
name: _miquplus-midnight-70b
merge_method: task_arithmetic
parameters:
normalize : false
weight: 1
models:
- model: meta-llama/Llama-2-70b-hf
- model: 152334H/miqu-1-70b-sf
- model: sophosympatheia/Midnight-Rose-70B-v2.0.3
base_model: meta-llama/Llama-2-70b-hf
dtype: float16
---
name: miquplus-midnight-70b
merge_method: linear
models:
- model: 152334H/miqu-1-70b-sf
parameters:
weight:
- filter: v_proj
value: [1, 1, 0, 0, 0, 0, 0, 0, 0, 1, 1]
- filter: o_proj
value: [1, 1, 0, 0, 0, 0, 0, 0, 0, 1, 1]
- filter: up_proj
value: [1, 1, 0, 0, 0, 0, 0, 0, 0, 1, 1]
- filter: gate_proj
value: [1, 1, 0, 0, 0, 0, 0, 0, 0, 1, 1]
- filter: down_proj
value: [1, 1, 0, 0, 0, 0, 0, 0, 0, 1, 1]
- value: 1
- model: _miquplus-midnight-70b
parameters:
weight:
- filter: v_proj
value: [0, 0, 1, 1, 1, 1, 1, 1, 1, 0, 0]
- filter: o_proj
value: [0, 0, 1, 1, 1, 1, 1, 1, 1, 0, 0]
- filter: up_proj
value: [0, 0, 1, 1, 1, 1, 1, 1, 1, 0, 0]
- filter: gate_proj
value: [0, 0, 1, 1, 1, 1, 1, 1, 1, 0, 0]
- filter: down_proj
value: [0, 0, 1, 1, 1, 1, 1, 1, 1, 0, 0]
- value: 0
base_model: 152334H/miqu-1-70b-sf
tokenizer_source: base
dtype: float16
NOTE
: Run with
mergekit-mega
rather than
mergekit
as there are 2 documents in this file.
Model background
Created using Mergekit in two stages:
-
First, a (broken!) "donor" model was created by adding (
Midnight-Rose-70B-v2.0.3-Llama-2-70b-hf) tomiqu-1-70b-sfusingtask_arithmetic. -
In the second stage, the
v_proj,o_proj,up_proj,gate_projanddown_projtensors were merged back intomiqu-1-70b-sf.
NOTE : After the second stage model is created the "donor" model can be deleted as it won't be needed again.
Click to see more details on the reasoning behind the merge
A. There are 3 very compelling reasons to keep the
q_proj
and
k_proj
matrices intact:
1. These are the only matrices that have the RoPE rotations applied:
see:
https://adalkiran.github.io/llama-nuts-and-bolts/#model-diagram
and trying to blend the the
q_proj
and
k_proj
matrices of models with 100x different base RoPE frequencies is just going to contract and dilate their perceived "distance" / "time" in the text they write (so both are wrong). If you did want to insist on blending them, then the base RoPE frequency would need to be set to something in-between that is the "least wrong" for both base RoPE frequencies and the maximum context reduced accordingly...
2. They don't actually appear to be responsible for domain-specific adaptation anyway:
3: The top ~30% of embedding dimensions for the 10k base RoPE frequency models aren't being used properly:
For LLaMA2(Touvron et al., 2023b), the critical dimension dextra is 92. This implies that only the first 92 dimensions of the qt, ks vectors of LLaMA2 have seen the complete positional information during the pre-training phase and are adequately trained. In other words, the last 36 dimensions lack sufficient training, contributing to the extrapolation challenges seen in RoPE-based LLMs
see: Scaling Laws of RoPE-based Extrapolation
So even if we could try to map the 10k RoPE models embedding vectors to match the 1M RoPE models embedding vectors post-dot-product, the 1M RoPE models embedding vectors are likely fine-tuned to use these top 36 dimensions properly and have thus changed very significantly during continued pre-training.
B. The norm vectors from the "donor" model can't be merged:
The
input_layernorm.weight
,
post_attention_layernorm.weight
and
norm.weight
vectors all use
LayerNorm
, and:
weight – the learnable weights of the module of shape normalized_shapenormalized_shape when elementwise_affine is set to True. The values are initialized to 1.
So to combine these we would need to use the
Geometric Mean
for the
task_arithmetic
operation (and hence why the "donor" model is broken).
C. The first and last 8 layers, the
embed_tokens.weight
and
lm_head.weight
tensors are not merged:
The model still functions if these are merged into the second stage model, but empirically it seems to work better if we don't do this as keep the original
miqu-1-70b-sf
weights.
Runs of jukofyork miquplus-midnight-70b on huggingface.co
20
Total runs
0
24-hour runs
0
3-day runs
0
7-day runs
0
30-day runs
More Information About miquplus-midnight-70b huggingface.co Model
Url of miquplus-midnight-70b
miquplus-midnight-70b huggingface.co Url
Provider of miquplus-midnight-70b huggingface.co
Other API from jukofyork
Updated:March 17 2025
Total runs:
1.4K
Run Growth:
0
Growth Rate:
0.00%
Total runs:
956
Run Growth:
0
Growth Rate:
0.00%
Total runs:
337
Run Growth:
0
Growth Rate:
0.00%
Total runs:
337
Run Growth:
0
Growth Rate:
0.00%
Total runs:
315
Run Growth:
-86
Growth Rate:
-27.30%
Total runs:
283
Run Growth:
0
Growth Rate:
0.00%
Total runs:
218
Run Growth:
0
Growth Rate:
0.00%
Total runs:
208
Run Growth:
80
Growth Rate:
38.46%
Total runs:
202
Run Growth:
0
Growth Rate:
0.00%
Total runs:
150
Run Growth:
91
Growth Rate:
60.67%
Total runs:
139
Run Growth:
0
Growth Rate:
0.00%
Total runs:
137
Run Growth:
88
Growth Rate:
64.23%
Total runs:
133
Run Growth:
133
Growth Rate:
100.00%
Total runs:
117
Run Growth:
0
Growth Rate:
0.00%
Total runs:
112
Run Growth:
69
Growth Rate:
61.61%
Total runs:
110
Run Growth:
0
Growth Rate:
0.00%
Total runs:
89
Run Growth:
63
Growth Rate:
70.79%
Total runs:
89
Run Growth:
40
Growth Rate:
44.94%
Total runs:
89
Run Growth:
-22
Growth Rate:
-24.72%
Total runs:
81
Run Growth:
67
Growth Rate:
82.72%
Total runs:
80
Run Growth:
0
Growth Rate:
0.00%
Total runs:
79
Run Growth:
0
Growth Rate:
0.00%
Total runs:
79
Run Growth:
0
Growth Rate:
0.00%
Total runs:
69
Run Growth:
0
Growth Rate:
0.00%
Total runs:
63
Run Growth:
0
Growth Rate:
0.00%
Total runs:
57
Run Growth:
42
Growth Rate:
73.68%
Total runs:
48
Run Growth:
0
Growth Rate:
0.00%
Total runs:
48
Run Growth:
36
Growth Rate:
75.00%
Total runs:
48
Run Growth:
0
Growth Rate:
0.00%
Total runs:
46
Run Growth:
0
Growth Rate:
0.00%
Total runs:
40
Run Growth:
0
Growth Rate:
0.00%
Total runs:
40
Run Growth:
30
Growth Rate:
75.00%
Total runs:
39
Run Growth:
0
Growth Rate:
0.00%
Total runs:
38
Run Growth:
0
Growth Rate:
0.00%
Total runs:
36
Run Growth:
0
Growth Rate:
0.00%
Total runs:
36
Run Growth:
0
Growth Rate:
0.00%
Total runs:
34
Run Growth:
34
Growth Rate:
100.00%
Total runs:
33
Run Growth:
26
Growth Rate:
78.79%
Total runs:
33
Run Growth:
0
Growth Rate:
0.00%
Total runs:
30
Run Growth:
25
Growth Rate:
83.33%
Total runs:
30
Run Growth:
0
Growth Rate:
0.00%
Total runs:
28
Run Growth:
0
Growth Rate:
0.00%
Total runs:
27
Run Growth:
15
Growth Rate:
55.56%
Total runs:
24
Run Growth:
24
Growth Rate:
100.00%
Total runs:
23
Run Growth:
0
Growth Rate:
0.00%
Total runs:
23
Run Growth:
0
Growth Rate:
0.00%
Total runs:
21
Run Growth:
0
Growth Rate:
0.00%
Total runs:
20
Run Growth:
0
Growth Rate:
0.00%
Total runs:
20
Run Growth:
13
Growth Rate:
65.00%
Total runs:
20
Run Growth:
8
Growth Rate:
40.00%
Total runs:
19
Run Growth:
0
Growth Rate:
0.00%
Total runs:
18
Run Growth:
9
Growth Rate:
50.00%
Total runs:
16
Run Growth:
0
Growth Rate:
0.00%
Total runs:
16
Run Growth:
0
Growth Rate:
0.00%
Total runs:
14
Run Growth:
0
Growth Rate:
0.00%
Total runs:
14
Run Growth:
0
Growth Rate:
0.00%
Total runs:
14
Run Growth:
8
Growth Rate:
57.14%
Total runs:
13
Run Growth:
0
Growth Rate:
0.00%
Total runs:
13
Run Growth:
11
Growth Rate:
84.62%
Total runs:
12
Run Growth:
0
Growth Rate:
0.00%
Total runs:
12
Run Growth:
0
Growth Rate:
0.00%
Total runs:
11
Run Growth:
-46
Growth Rate:
-418.18%
Total runs:
10
Run Growth:
0
Growth Rate:
0.00%
Total runs:
10
Run Growth:
3
Growth Rate:
30.00%
Total runs:
9
Run Growth:
3
Growth Rate:
33.33%
Total runs:
9
Run Growth:
7
Growth Rate:
77.78%
Total runs:
9
Run Growth:
5
Growth Rate:
55.56%
Total runs:
7
Run Growth:
0
Growth Rate:
0.00%
Total runs:
5
Run Growth:
0
Growth Rate:
0.00%
Total runs:
5
Run Growth:
0
Growth Rate:
0.00%
Total runs:
4
Run Growth:
0
Growth Rate:
0.00%
Total runs:
4
Run Growth:
0
Growth Rate:
0.00%
Total runs:
4
Run Growth:
0
Growth Rate:
0.00%
Total runs:
4
Run Growth:
0
Growth Rate:
0.00%
Total runs:
3
Run Growth:
0
Growth Rate:
0.00%
Total runs:
2
Run Growth:
0
Growth Rate:
0.00%
Total runs:
1
Run Growth:
0
Growth Rate:
0.00%
Total runs:
0
Run Growth:
0
Growth Rate:
0.00%
Total runs:
0
Run Growth:
0
Growth Rate:
0.00%
Total runs:
0
Run Growth:
0
Growth Rate:
0.00%
Total runs:
0
Run Growth:
0
Growth Rate:
0.00%
Total runs:
0
Run Growth:
0
Growth Rate:
0.00%
Total runs:
0
Run Growth:
0
Growth Rate:
0.00%
Total runs:
0
Run Growth:
0
Growth Rate:
0.00%
Total runs:
0
Run Growth:
0
Growth Rate:
0.00%