Semantic segmentation for automotive street scenes
FFNet-78S is a "fuss-free network" that segments street scene images with per-pixel classes like road, sidewalk, and pedestrian. Trained on the Cityscapes dataset.
This model is an implementation of FFNet-78S found
here
.
This repository provides scripts to run FFNet-78S on Qualcomm® devices.
More details on model performance across various devices, can be found
here
.
Model Details
Model Type:
Semantic segmentation
Model Stats:
Model checkpoint: ffnet78S_dBBB_cityscapes_state_dict_quarts
The package contains a simple end-to-end demo that downloads pre-trained
weights and runs this model on a sample input.
python -m qai_hub_models.models.ffnet_78s.demo
The above demo runs a reference implementation of pre-processing, model
inference, and post processing.
NOTE
: If you want running in a Jupyter Notebook or Google Colab like
environment, please add the following to your cell (instead of the above).
%run -m qai_hub_models.models.ffnet_78s.demo
Run model on a cloud-hosted device
In addition to the demo, you can also run the model on a cloud-hosted Qualcomm®
device. This script does the following:
Performance check on-device on a cloud-hosted device
Downloads compiled assets that can be deployed on-device for Android.
Accuracy check between PyTorch and on-device outputs.
python -m qai_hub_models.models.ffnet_78s.export
Profile Job summary of FFNet-78S
--------------------------------------------------
Device: SA8255 (Proxy) (13)
Estimated Inference Time: 23.75 ms
Estimated Peak Memory Range: 23.07-40.26 MB
Compute Units: NPU (235) | Total (235)
How does this work?
This
export script
leverages
Qualcomm® AI Hub
to optimize, validate, and deploy this model
on-device. Lets go through each step below in detail:
Step 1:
Compile model for on-device deployment
To compile a PyTorch model for on-device deployment, we first trace the model
in memory using the
jit.trace
and then call the
submit_compile_job
API.
import torch
import qai_hub as hub
from qai_hub_models.models.ffnet_78s import Model
# Load the model
torch_model = Model.from_pretrained()
# Device
device = hub.Device("Samsung Galaxy S23")
# Trace model
input_shape = torch_model.get_input_spec()
sample_inputs = torch_model.sample_inputs()
pt_model = torch.jit.trace(torch_model, [torch.tensor(data[0]) for _, data in sample_inputs.items()])
# Compile model on a specific device
compile_job = hub.submit_compile_job(
model=pt_model,
device=device,
input_specs=torch_model.get_input_spec(),
)
# Get target model to run on-device
target_model = compile_job.get_target_model()
Step 2:
Performance profiling on cloud-hosted device
After compiling models from step 1. Models can be profiled model on-device using the
target_model
. Note that this scripts runs the model on a device automatically
provisioned in the cloud. Once the job is submitted, you can navigate to a
provided job URL to view a variety of on-device performance metrics.
FFNet-78S huggingface.co is an AI model on huggingface.co that provides FFNet-78S's model effect (), which can be used instantly with this qualcomm FFNet-78S model. huggingface.co supports a free trial of the FFNet-78S model, and also provides paid use of the FFNet-78S. Support call FFNet-78S model through api, including Node.js, Python, http.
FFNet-78S huggingface.co is an online trial and call api platform, which integrates FFNet-78S's modeling effects, including api services, and provides a free online trial of FFNet-78S, you can try FFNet-78S online for free by clicking the link below.
qualcomm FFNet-78S online free url in huggingface.co:
FFNet-78S is an open source model from GitHub that offers a free installation service, and any user can find FFNet-78S on GitHub to install. At the same time, huggingface.co provides the effect of FFNet-78S install, users can directly use FFNet-78S installed effect in huggingface.co for debugging and trial. It also supports api for free installation.