Add 6 more layers (16 to 21) with identical structure to existing layers
Embedding Layer
model.embed_tokens.weight
model.embed_tokens.weight
Already identical
Self-Attention Layers
22 sets of (q_proj, k_proj, v_proj, o_proj) weights
16 sets of (q_proj, k_proj, v_proj, o_proj) weights
Add 6 more sets of self-attention weights
MLP Layers
22 sets of (gate_proj, up_proj, down_proj) weights
16 sets of (gate_proj, up_proj, down_proj) weights
Add 6 more sets of MLP weights
Layer Normalization
22 sets of (input_layernorm, post_attention_layernorm) weights
16 sets of (input_layernorm, post_attention_layernorm) weights
Add 6 more sets of layer normalization weights
Final Normalization
model.norm.weight
model.norm.weight
Already identical
Language Model Head
lm_head.weight
lm_head.weight
Already identical
Layer Structure
Consistent across all 22 layers
Consistent across all 16 layers
Maintain the same structure when adding new layers
Hidden Size
Likely consistent (inferred from weight names)
Likely consistent (inferred from weight names)
Ensure new layers use the same hidden size
Attention Heads
Likely consistent (inferred from weight names)
Likely consistent (inferred from weight names)
Ensure new layers use the same number of attention heads
Intermediate MLP Size
Likely consistent (inferred from weight names)
Likely consistent (inferred from weight names)
Ensure new layers use the same intermediate MLP size
Position Embeddings
Not explicitly mentioned (might be part of embed_tokens)
Not explicitly mentioned (might be part of embed_tokens)
Ensure position embeddings support the maximum sequence length of the first model
Vocabulary Size
Determined by embed_tokens and lm_head dimensions
Determined by embed_tokens and lm_head dimensions
Already identical (assuming dimensions match)
further investigation
there is not differences between these models but for some reason i'm constantly facing this error when doing passthrough:
Traceback (most recent call last):
File "/home/zeus/miniconda3/envs/cloudspace/bin/mergekit-yaml", line 8, in <module>
sys.exit(main())
File "/home/zeus/miniconda3/envs/cloudspace/lib/python3.10/site-packages/click/core.py", line 1157, in __call__
return self.main(*args, **kwargs)
File "/home/zeus/miniconda3/envs/cloudspace/lib/python3.10/site-packages/click/core.py", line 1078, in main
rv = self.invoke(ctx)
File "/home/zeus/miniconda3/envs/cloudspace/lib/python3.10/site-packages/click/core.py", line 1434, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/home/zeus/miniconda3/envs/cloudspace/lib/python3.10/site-packages/click/core.py", line 783, in invoke
return __callback(*args, **kwargs)
File "/teamspace/studios/this_studio/mergekit/mergekit/options.py", line 82, in wrapper
f(*args, **kwargs)
File "/teamspace/studios/this_studio/mergekit/mergekit/scripts/run_yaml.py", line 47, in main
run_merge(
File "/teamspace/studios/this_studio/mergekit/mergekit/merge.py", line 96, in run_merge
for _task, value in exec.run(quiet=options.quiet):
File "/teamspace/studios/this_studio/mergekit/mergekit/graph.py", line 197, in run
res = task.execute(**arguments)
File "/teamspace/studios/this_studio/mergekit/mergekit/io/tasks.py", line 86, in execute
raise RuntimeError(
RuntimeError: Tensor lm_head.weight required but not present in model meta-llama/Llama-3.2-1B
which seems odd given the following output layers from llama-3-1b:
llama-3-1b huggingface.co is an AI model on huggingface.co that provides llama-3-1b's model effect (), which can be used instantly with this appvoid llama-3-1b model. huggingface.co supports a free trial of the llama-3-1b model, and also provides paid use of the llama-3-1b. Support call llama-3-1b model through api, including Node.js, Python, http.
llama-3-1b huggingface.co is an online trial and call api platform, which integrates llama-3-1b's modeling effects, including api services, and provides a free online trial of llama-3-1b, you can try llama-3-1b online for free by clicking the link below.
appvoid llama-3-1b online free url in huggingface.co:
llama-3-1b is an open source model from GitHub that offers a free installation service, and any user can find llama-3-1b on GitHub to install. At the same time, huggingface.co provides the effect of llama-3-1b install, users can directly use llama-3-1b installed effect in huggingface.co for debugging and trial. It also supports api for free installation.
