Clip As Service

🏄 Embed/reason/rank images and sentences with CLIP models
Alternatives To Clip As Service
Project NameStarsDownloadsRepos Using ThisPackages Using ThisMost Recent CommitTotal ReleasesLatest ReleaseOpen IssuesLicenseLanguage
Transformers102,6136491120 hours ago91June 21, 2022724apache-2.0Python
🤗 Transformers: State-of-the-art Machine Learning for Pytorch, TensorFlow, and JAX.
Stable Diffusion Webui80,323
19 hours ago1,928agpl-3.0Python
Stable Diffusion web UI
Pytorch67,46314619 hours ago23August 10, 202212,145otherPython
Tensors and Dynamic neural networks in Python with strong GPU acceleration
Real Time Voice Cloning41,693
a month ago129otherPython
Clone a voice in 5 seconds to generate arbitrary speech in real-time
Yolov539,026
19 hours ago35May 21, 2022276agpl-3.0Python
YOLOv5 🚀 in PyTorch > ONNX > CoreML > TFLite
Made With Ml33,193
a month ago5May 15, 201911mitJupyter Notebook
Learn how to responsibly develop, deploy and maintain production machine learning applications.
Mockingbird28,5432a month ago9February 28, 2022411otherPython
🚀AI拟声: 5秒内克隆您的声音并生成任意语音内容 Clone a voice in 5 seconds to generate arbitrary speech in real-time
Gfpgan27,65112 months ago11February 15, 2022209otherPython
GFPGAN aims at developing Practical Algorithms for Real-world Face Restoration.
Pytorch Tutorial26,129
2 months ago85mitPython
PyTorch Tutorial for Deep Learning Researchers
Ray25,8638019919 hours ago76June 09, 20222,888apache-2.0Python
Ray is a unified framework for scaling AI and Python applications. Ray consists of a core distributed runtime and a toolkit of libraries (Ray AIR) for accelerating ML workloads.
Alternatives To Clip As Service
Select To Compare


Alternative Project Comparisons
Readme




CLIP-as-service logo: The data structure for unstructured data


Embedding image and sentence into fixed-length vectors via CLIP

Python 3.7 3.8 3.9 3.10 PyPI

CLIP-as-service is a low-latency high-scalability embedding service for images and texts. It can be easily integrated as a microservice into neural search solutions.

⚡ Fast: Serve CLIP models with ONNX runtime and PyTorch JIT with 800QPS[*]. Non-blocking duplex streaming on requests and responses, designed for large data and long-running tasks.

🫐 Elastic: Horizontally scale up and down multiple CLIP models on single GPU, with automatic load balancing.

🐥 Easy-to-use: No learning curve, minimalist design on client and server. Intuitive and consistent API for image and sentence embedding.

👒 Modern: Async client support. Easily switch between gRPC, HTTP, Websocket protocols with TLS and compressions.

🍱 Integration: Smoothly integrated with neural search ecosystem including Jina and DocArray. Build cross-modal and multi-modal solution in no time.

[*] with default config (single replica, PyTorch no JIT) on GeForce RTX 3090.

Documentation

Install

CLIP-as-service consists of two Python packages clip-server and clip-client that can be installed independently. Both require Python 3.7+.

Install server

pip install clip-server

To run CLIP model via ONNX (default is via PyTorch):

pip install "clip-server[onnx]"

Install client

pip install clip-client

Quick check

You can run a simple connectivity check after install.

C/S Command Expect output
Server
python -m clip_server
Expected server output
Client
from clip_client import Client

c = Client('grpc://0.0.0.0:23456')
c.profile()
Expected clip-client output

You can change 0.0.0.0 to the intranet or public IP address to test the connectivity over private and public network. If you encounter some errors, please find the solution here.

Get Started

Basic usage

  1. Start the server: python -m clip_server. Remember its address and port.
  2. Create a client:
     from clip_client import Client
    
     c = Client('grpc://87.191.159.105:51000')
    
  3. To get sentence embedding:
    r = c.encode(['First do it', 'then do it right', 'then do it better'])
    
    print(r.shape)  # [3, 512] 
    
  4. To get image embedding:
    r = c.encode(['apple.png',  # local image 
                  'https://docarray.jina.ai/_static/favicon.png',  # remote image
                  'data:image/gif;base64,R0lGODlhEAAQAMQAAORHHOVSKudfOulrSOp3WOyDZu6QdvCchPGolfO0o/XBs/fNwfjZ0frl3/zy7////wAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAACH5BAkAABAALAAAAAAQABAAAAVVICSOZGlCQAosJ6mu7fiyZeKqNKToQGDsM8hBADgUXoGAiqhSvp5QAnQKGIgUhwFUYLCVDFCrKUE1lBavAViFIDlTImbKC5Gm2hB0SlBCBMQiB0UjIQA7'])  # in image URI
    
    print(r.shape)  # [3, 512]
    

More comprehensive server & client configs can be found in the docs.

Text-to-image cross-modal search in 10 Lines

Let's build a text-to-image search using CLIP-as-service. Namely, user input a sentence and the program returns the matched images. We will use Totally Looks Like dataset and DocArray package. Note that DocArray is included within clip-client as an upstream dependency, so you don't need to install it separately.

Load images

First we load images. You can simply pull it from Jina Cloud:

from docarray import DocumentArray

da = DocumentArray.pull('ttl-original', show_progress=True, local_cache=True)
or download TTL dataset, unzip, load manually

Alternatively, you can go to Totally Looks Like official website, unzip and load images as follows:

from docarray import DocumentArray

da = DocumentArray.from_files(['left/*.jpg', 'right/*.jpg'])

The dataset contains 12,032 images, hence it may take half minute to pull. Once done, you can visualize it and get the first taste of those images.

da.plot_image_sprites()

Visualize of the image sprite of Totally looks like dataset

Encode images

Start the server with python -m clip_server. Say it is at 87.191.159.105:51000 with GRPC protocol (you will get this information after running the server).

Create a Python client script:

from clip_client import Client

c = Client(server='grpc://87.191.159.105:51000')

da = c.encode(da, show_progress=True)

Depending on your GPU and client-server network, it could take a while to embed 12K images. In my case, it takes ~2 minute.

Download the pre-encoded dataset

For people who are impatient or lack of GPU, waiting can be a hell. In this case, you can simply pull our pre-encoded image dataset.

from docarray import DocumentArray

da = DocumentArray.pull('ttl-embedding', show_progress=True, local_cache=True)

Search via sentence

Let's build a simple prompt to allow user to type sentence:

while True:
    vec = c.encode([input('sentence> ')])
    r = da.find(query=vec, limit=9)
    r.plot_image_sprites()

Showcase

Now you can input arbitrary English sentences and view the top-9 matched images. Search is fast and instinct. Let's have some fun:

"a happy potato" "a super evil AI" "a guy enjoying his burger"

Visualize of the image sprite of Totally looks like dataset

Visualize of the image sprite of Totally looks like dataset

Visualize of the image sprite of Totally looks like dataset

"professor cat is very serious" "an ego engineer lives with parent" "there will be no tomorrow so lets eat unhealthy"

Visualize of the image sprite of Totally looks like dataset

Visualize of the image sprite of Totally looks like dataset

Visualize of the image sprite of Totally looks like dataset

Let's save the embedding result for our next example:

da.save_binary('ttl-image')

Image-to-text cross-modal search in 10 Lines

We can also switch the input and output of the last program to achieve image-to-text search. Precisely, given a query image find the sentence that best describes the image.

Let's use all sentences from the book "Pride and Prejudice".

from docarray import Document, DocumentArray

d = Document(uri='https://www.gutenberg.org/files/1342/1342-0.txt').load_uri_to_text()
da = DocumentArray(
    Document(text=s.strip()) for s in d.text.replace('\r\n', '').split('.') if s.strip()
)

Let's look at what we got:

da.summary()
            Documents Summary            
                                         
  Length                 6403            
  Homogenous Documents   True            
  Common Attributes      ('id', 'text')  
                                         
                     Attributes Summary                     
                                                            
  Attribute   Data type   #Unique values   Has empty value  
 ────────────────────────────────────────────────────────── 
  id          ('str',)    6403             False            
  text        ('str',)    6030             False            

Encode sentences

Now encode these 6403 sentences, it may take 10s or less depending on your GPU and network:

from clip_client import Client

c = Client('grpc://87.191.159.105:51000')

r = c.encode(da, show_progress=True)
Download the pre-encoded dataset

Again, for people who are impatient or lack of GPU, we have prepared a pre-encoded text dataset.

from docarray import DocumentArray

da = DocumentArray.pull('ttl-textual', show_progress=True, local_cache=True)

Search via image

Let's load our previously stored image embedding; randomly sample image Document from it, then find top-1 nearest neighbour of each.

from docarray import DocumentArray

img_da = DocumentArray.load_binary('ttl-image')

for d in img_da.sample(10):
    print(da.find(d.embedding, limit=1)[0].text)

Showcase

Fun time! Note, unlike the previous example, here the input is an image, the sentence is the output. All sentences come from the book "Pride and Prejudice".

Visualize of the image sprite of Totally looks like dataset

Visualize of the image sprite of Totally looks like dataset

Visualize of the image sprite of Totally looks like dataset

Visualize of the image sprite of Totally looks like dataset

Visualize of the image sprite of Totally looks like dataset

Besides, there was truth in his looks Gardiner smiled what’s his name By tea time, however, the dose had been enough, and Mr You do not look well

Visualize of the image sprite of Totally looks like dataset

Visualize of the image sprite of Totally looks like dataset

Visualize of the image sprite of Totally looks like dataset

Visualize of the image sprite of Totally looks like dataset

Visualize of the image sprite of Totally looks like dataset

“A gamester!” she cried If you mention my name at the Bell, you will be attended to Never mind Miss Lizzy’s hair Elizabeth will soon be the wife of Mr I saw them the night before last

Intrigued? That's only scratching the surface of what CLIP-as-service is capable of. Read our docs to learn more.

Support

Join Us

CLIP-as-service is backed by Jina AI and licensed under Apache-2.0. We are actively hiring AI engineers, solution engineers to build the next neural search ecosystem in open-source.

Popular Deep Learning Projects
Popular Pytorch Projects
Popular Machine Learning Categories
Related Searches

Get A Weekly Email With Trending Projects For These Categories
No Spam. Unsubscribe easily at any time.
Python
Deep Learning
Pytorch
Onnx