# DeTi*k*Zify<br><sub><sup>Synthesizing Graphics Programs for Scientific Figures and Sketches with Ti*k*Z</sup></sub>
[![Hugging Face](https://img.shields.io/badge/View%20on%20Hugging%20Face-blue?labelColor=gray&logo=data:image/png;base64, )](<redacted for anonymity>)
[![Colab](https://colab.research.google.com/assets/colab-badge.svg)](<redacted for anonymity>)

Creating high-quality scientific figures can be time-consuming and challenging,
even though sketching ideas on paper is relatively easy. Furthermore,
recreating existing figures that are not stored in formats preserving semantic
information is equally complex. To tackle this problem, we introduce
[DeTi*k*Zify](<redacted for anonymity>), a novel multimodal
language model that automatically synthesizes scientific figures as
semantics-preserving [Ti*k*Z](https://github.com/pgf-tikz/pgf) graphics
programs based on sketches and existing figures. We also introduce an
MCTS-based inference algorithm that enables DeTi*k*Zify to iteratively refine
its outputs without the need for additional training.

## Installation
The Python package of DeTi*k*Zify can be easily installed using
[pip](https://pip.pypa.io/en/stable):
```sh
pip install 'detikzify @ git+<redacted for anonymity>'
```
Or, if your goal is to run the included [examples](examples), clone the
repository and install it in editable mode like this:
 ```sh
git clone "<redacted for anonymity>"
pip install -e DeTikZify
 ```
In addition, DeTi*k*Zify requires a full
[TeX Live 2023](https://www.tug.org/texlive) installation,
[ghostscript](https://www.ghostscript.com), and,
[poppler](https://poppler.freedesktop.org) which you have to install through
your package manager or via other means.

> [!TIP]
> If you encounter difficulties with installation and inference on your own
> hardware, consider using our demo on [Google
> Colab](https://colab.research.google.com/drive/1hPWqucbPGTavNlYvOBvSNBAwdcPZKe8F).
> Please note, however, that setting up the environment may take some time, and
> the free tier is only applicable for inference of the 1b models. For a more
> convenient experience and access to all models, consider duplicating our
> [Hugging Face Space](<redacted for anonymity>) with a
> paid private GPU runtime.

## Usage
If all required dependencies are installed, the full range of DeTi*k*Zify
features such as compiling, rendering, and saving Ti*k*Z graphics, and
MCTS-based inference can be accessed through its programming interface:
```python
from operator import itemgetter

from detikzify.model import load
from detikzify.infer import DetikzifyPipeline
import torch

image = "https://w.wiki/A7Cc"
pipeline = DetikzifyPipeline(*load(
    base_model="<redacted for anonymity>",
    device_map="auto",
    torch_dtype=torch.bfloat16,
))

# generate a single TikZ program
fig = pipeline.sample(image=image)

# if it compiles, rasterize it and show it
if fig.is_rasterizable:
    fig.rasterize().show()

# run MCTS for 10 minutes and generate multiple TikZ programs
figs = set()
for score, fig in pipeline.simulate(image=image, timeout=600):
    figs.add((score, fig))

# save the best TikZ program
best = sorted(figs, key=itemgetter(0))[-1][1]
best.save("fig.tex")
```
For interactive use and additional usage tips, we recommend checking out our
[web UI](detikzify/webui), which can be started from the command line (use
`--help` for a list of all options):
```sh
python -m detikzify.webui --light
```
More involved examples, for example for evaluation and training, can be found
in the [examples](examples) folder.

## Model Weights & Datasets
We upload all our models and datasets to the [Hugging Face
Hub](<redacted for anonymity>).
However, please note that for the public release of the DaTi*k*Z<sub>v2</sub>
dataset, we had to remove a considerable portion of Ti*k*Z drawings originating
from [arXiv](https://arxiv.org), as the [arXiv non-exclusive
license](https://arxiv.org/licenses/nonexclusive-distrib/1.0/license.html) does
not permit redistribution. We do, however, release our [dataset creation
scripts](<redacted for anonymity>) and encourage anyone to recreate
the full version of DaTi*k*Z<sub>v2</sub> themselves.

## Acknowledgments
The implementation of the model architecture is largely based on
[LLaVA](https://github.com/haotian-liu/LLaVA). Our MCTS implementation takes
heavy inspiration from
[VMCTS](https://github.com/namin/llm-verified-with-monte-carlo-tree-search).
