Locutusque / Hyperion-3.0-Mixtral-3x7B

huggingface.co
Total runs: 75
24-hour runs: 0
7-day runs: 2
30-day runs: 13
Model's Last Updated: March 16 2024
text-generation

Introduction of Hyperion-3.0-Mixtral-3x7B

Model Details of Hyperion-3.0-Mixtral-3x7B

Hyperion-3.0-Mixtral-3x7B

Model Details

This is an experimental first attempt at creating a Mixture of Experts (MoE) language model by combining several Mistral expert models. The model uses the hyperion-3.0-beta architecture as the base, with a bfloat16 output dtype. The gating mechanism is set to hidden and two experts are consulted per token ( experts_per_token: 2 ).

The model incorporates three expert models:

  1. hyperion-3.0-beta : Focused on science, math, and coding tasks
  2. dibt-mistral-7b : Handles open-ended questions, summarization, and stream of consciousness.
  3. rp-mistral-7b : Specializes in roleplaying and character-based conversations

Each expert is trained on a set of positive and negative prompts to guide its specialization.

Intended Use and Limitations

This MoE model is an early prototype and may not exhibit optimal performance. It is intended for research and experimentation purposes only, and should not be used in production environments or for critical applications.

Please note that the expert models mentioned in the configuration have not been publicly released yet. They are expected to be made available in the near future, at which point this MoE model can be fully instantiated and evaluated.

Training Details

The base model and experts were trained using QLoRA and SFT. However, the specific details of the training data, hyperparameters, and optimization techniques used for this MoE model are not available at this time.

Feedback and Future Updates

As this is an experimental model, feedback and suggestions are welcome. Future updates may include improvements to the gating mechanism, fine-tuning of the expert models, and the incorporation of additional experts to enhance the model's performance and breadth of knowledge.

Runs of Locutusque Hyperion-3.0-Mixtral-3x7B on huggingface.co

75
Total runs
0
24-hour runs
0
3-day runs
2
7-day runs
13
30-day runs

More Information About Hyperion-3.0-Mixtral-3x7B huggingface.co Model

More Hyperion-3.0-Mixtral-3x7B license Visit here:

https://choosealicense.com/licenses/apache-2.0

Hyperion-3.0-Mixtral-3x7B huggingface.co

Hyperion-3.0-Mixtral-3x7B huggingface.co is an AI model on huggingface.co that provides Hyperion-3.0-Mixtral-3x7B's model effect (), which can be used instantly with this Locutusque Hyperion-3.0-Mixtral-3x7B model. huggingface.co supports a free trial of the Hyperion-3.0-Mixtral-3x7B model, and also provides paid use of the Hyperion-3.0-Mixtral-3x7B. Support call Hyperion-3.0-Mixtral-3x7B model through api, including Node.js, Python, http.

Hyperion-3.0-Mixtral-3x7B huggingface.co Url

https://huggingface.co/Locutusque/Hyperion-3.0-Mixtral-3x7B

Locutusque Hyperion-3.0-Mixtral-3x7B online free

Hyperion-3.0-Mixtral-3x7B huggingface.co is an online trial and call api platform, which integrates Hyperion-3.0-Mixtral-3x7B's modeling effects, including api services, and provides a free online trial of Hyperion-3.0-Mixtral-3x7B, you can try Hyperion-3.0-Mixtral-3x7B online for free by clicking the link below.

Locutusque Hyperion-3.0-Mixtral-3x7B online free url in huggingface.co:

https://huggingface.co/Locutusque/Hyperion-3.0-Mixtral-3x7B

Hyperion-3.0-Mixtral-3x7B install

Hyperion-3.0-Mixtral-3x7B is an open source model from GitHub that offers a free installation service, and any user can find Hyperion-3.0-Mixtral-3x7B on GitHub to install. At the same time, huggingface.co provides the effect of Hyperion-3.0-Mixtral-3x7B install, users can directly use Hyperion-3.0-Mixtral-3x7B installed effect in huggingface.co for debugging and trial. It also supports api for free installation.

Hyperion-3.0-Mixtral-3x7B install url in huggingface.co:

https://huggingface.co/Locutusque/Hyperion-3.0-Mixtral-3x7B

Url of Hyperion-3.0-Mixtral-3x7B

Hyperion-3.0-Mixtral-3x7B huggingface.co Url

Provider of Hyperion-3.0-Mixtral-3x7B huggingface.co

Locutusque
ORGANIZATIONS

Other API from Locutusque