Jekyll2023-12-02T09:05:53+00:00http://alexene.dev/feed.xmlAlexandru Ene blogAlexandru EneHaving fun with DALL-E 22022-07-24T00:00:00+00:002022-07-24T00:00:00+00:00http://alexene.dev/2022/07/24/Having-fun-with-DALL-E-2<p>I have beta access to DALL-E 2 and I’ve played around with it this weekend. It’s incredible. However, there’s one <em>“problem”</em>. It requires interesting prompts to produce interesting images.<br />
Any creative project I start eventually suffers from the same issue. I’m not a creative person. I run out of ideas quite fast. It turns out this problem has an easy fix.</p>
<h2 id="intro">Intro</h2>
<p>We need to first understand how DALL-E works. More precisely how we use it, not how it actually works. Nobody really knows how machine learning works besides multiplying lots of matrices.</p>
<p>Today there’s no official API for DALL-E 2. You get a prompt bar where you can put the input text and you get out 4 images. You can also input an image and get variations on it. Image sizes are fixed and there’s a report button in case something goes horribly wrong. Easy!
<img src="/images/dalle/dalle2_site.png" alt="dalle2 site" /></p>
<p>You get some free tokens every month and generating a series of 4 images consumes one token. I’ve spent my initial ones on genrating images of cats doing things and Formula 1 cars. While images with cats are always cool and interesting, I can honestly say that I couldn’t generate interesting images of Formula 1 cars. I wanted to generate one for a friend but it’s just not possible. Please save your tokens and don’t even try. Even <em>“Formula 1 car made from jellybeans”</em> had dissapointing results. We can only hope DALL-E 3 will tackle the problem of interesting looking (by my standards) Formula 1 cars.</p>
<p>The limited free tokens are the least of my problems. You can always buy more, and the prices are relatively reasonable for the amount of fun provided. As I said before, my biggest problem is that I have zero creativity. DALL-E is successful in automating the creation of images from a given text, but this solves half of the problem – that I can’t draw. We need to go further. We also need to have a good title for our image. A good title is 90% of any creative work, just like a good variable name.</p>
<h2 id="generating-titles">Generating titles</h2>
<p>This means that we need to automate the generation of image titles (prompts). OpenAI, the creators of DALL-E already have our backs here, as they also have GPT3. This one is pretty good at doing things with text.</p>
<p>How would we go about generating descriptions of interesting images?
This is easy, we don’t even need to fine-tune a GPT3 model. We need to inject some examples in the prompts for GPT3.
The prompts that I used look like this:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>Generate a Title for a painting
Title: A cat reading a book, watercolour painting
Title: A cat drawing a picture of another cat, pencil drawing
Title: vacuum cleaner in a museum
Title: anger as abstract art
</code></pre></div></div>
<p>Then we make a request to GPT3’s completion API with the prompt above. For the model I’ve used the most beefy one <code class="language-plaintext highlighter-rouge">davinci-02</code>.
I set the temperature to <code class="language-plaintext highlighter-rouge">0.9</code> so it produces more funky stuff.
I get a few responses back (set them to 5), clean them up a bit, keep the interesting ones and put them in the title pool. Over time that title pool increases so everything that follows <code class="language-plaintext highlighter-rouge">Title: </code> is chosen as random from the list of previous titles I liked.</p>
<p>Here are some images that DALL-E made from GPT3-generated titles:</p>
<p><em>the first step, watercolor painting</em>
<img src="/images/dalle/the%20first%20step.png" alt="first step" /></p>
<p><em>an old man and his cat, oil painting</em>
<img src="/images/dalle/an%20old%20man%20an%20his%20cat.png" alt="man and his dog" /></p>
<p><em>A surrealist garden party</em>
<img src="/images/dalle/a%20sureallist%20garden%20party.png" alt="cat life" /></p>
<p><em>Colorful abstract art</em>
<img src="/images/dalle/colorful%20abstract%20art.png" alt="" /></p>
<p><em>a cat made of clouds in a garden party vaporwave</em>
<img src="/images/dalle/a%20cat%20made%20of%20clouds.png" alt="cat made of clouds" /></p>
<p><em>From wiskers to purrs, geometric art</em>
<img src="/images/dalle/from%20wiskers%20to%20purrs.png" alt="wiskers to purrs" /></p>
<p><em>my grandfather’s study</em>
<img src="/images/dalle/my%20grandfather's%20study.png" alt="gradnfather study" /></p>
<p><em>a cat’s life, acrylic painting</em>
<img src="/images/dalle/a%20cat%20life.png" alt="cat life" /></p>
<h2 id="conclusion">Conclusion</h2>
<p>I’m like an art curator, these two collaborate and generate things for me, and I just keep the ones I like. I’m sure there are better ways to go about this and further improve the title generation, but I’ve found quite a lot of enjoyment in discovering what they create.</p>
<p>There are some questions people may be interested in and that make up for good Medium articles such as:<br />
Can AIs be creative?<br />
Do humans have any unique traits left that a machine can’t simulate?<br />
Will we all live in pods surviving on disgusting soup?</p>
<p>I don’t know and I don’t care.<br />
In the meantime, I’ll sit back and enjoy watching these <em>“Cats in conversation”</em>:
<img src="/images/dalle/cats%20in%20conversation.png" alt="cats in conversation" /></p>Alexandru EneI have beta access to DALL-E 2 and I’ve played around with it this weekend. It’s incredible. However, there’s one “problem”. It requires interesting prompts to produce interesting images. Any creative project I start eventually suffers from the same issue. I’m not a creative person. I run out of ideas quite fast. It turns out this problem has an easy fix.Webassembly Without The Browser Part 22020-09-04T00:00:00+00:002020-09-04T00:00:00+00:00http://alexene.dev/2020/09/04/webassembly-without-the-browser-part-2<p>In <a href="https://alexene.dev/2020/08/17/webassembly-without-the-browser-part-1.html">part 1</a> we have learned how to set up WebAssembly VM to run a simple rust program that can add two numbers and print the result to stdout. In part 2 we will go over <strong>Debugging</strong> and <strong>Binary size</strong>.</p>
<h2 id="debugging">Debugging</h2>
<p>Debugging is one of the rough edges of WebAssembly. To understand why this is a rough edge, we must first have a high-level understanding on how a WebAssembly VM works. We can split them into two categories: WASM VMs with JIT and WASM VMs without JIT. Right now, debugging is possible only for JIT-enabled VM e.g.: <a href="https://github.com/bytecodealliance/wasmtime">Wasmtime</a>.</p>
<p>JIT is the key functionality that they use to enable a seamless debugging experience between the host program (the one that uses the VM) and the WebAssembly program. We usually use GDB or LLDB to debug the host programs. For example, to make this work, Wasmtime generates the JIT code, then it patches the debug info for the rust WebAssembly binary and calls a magical function named: <code class="language-plaintext highlighter-rouge">__jit_debug_register_code</code>. After this function is intercepted by GDB/LLDB and the JITed WebAssembly code can be debugged in the same session as the host program. It tells LLDB that this JIT-generated code has that patched debug information. Pretty neat!</p>
<p>In order for us to debug WebAssembly, there are a few key steps that we have to take:</p>
<ol>
<li>Use Visual Studio Code as our IDE (makes step 2 and 3 possible).</li>
<li>Install CodeLLDB plugin for VSCode. Other GDB/LLDB plugins are fine but this good on Linux/OSX/Windows too.</li>
<li>Tell Wasmtime to emit debug information. We use the configuration object (full program can be found in <a href="https://alexene.dev/2020/08/17/webassembly-without-the-browser-part-1.html">part 1</a> of this series): <code class="language-plaintext highlighter-rouge">let engine = Engine::new(Config::new().debug_info(true));</code></li>
<li>Set up Visual Studio Code <code class="language-plaintext highlighter-rouge">launch.json</code> as if you’d be debugging the host program. For example:
<div class="language-json highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="w"> </span><span class="nl">"configurations"</span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="w">
</span><span class="p">{</span><span class="w">
</span><span class="nl">"name"</span><span class="p">:</span><span class="w"> </span><span class="s2">"Debug WASM"</span><span class="p">,</span><span class="w">
</span><span class="nl">"type"</span><span class="p">:</span><span class="w"> </span><span class="s2">"lldb"</span><span class="p">,</span><span class="w">
</span><span class="nl">"request"</span><span class="p">:</span><span class="w"> </span><span class="s2">"launch"</span><span class="p">,</span><span class="w">
</span><span class="nl">"program"</span><span class="p">:</span><span class="w"> </span><span class="s2">"${workspaceFolder}/target/debug/wasm_host.exe"</span><span class="p">,</span><span class="w">
</span><span class="nl">"cwd"</span><span class="p">:</span><span class="w"> </span><span class="s2">"${workspaceFolder}"</span><span class="p">,</span><span class="w">
</span><span class="nl">"args"</span><span class="p">:</span><span class="w"> </span><span class="p">[],</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">]</span><span class="w">
</span></code></pre></div> </div>
</li>
</ol>
<p>Here is a screenshot of me debugging a WebAssembly program (on the right) and the host VM (on the left). As you can observe, the local variables are listed as expected and call stacks work just fine.</p>
<p><img src="/images/wasm/debug.png" alt="webAssembly debugging" /></p>
<p>If you’re deploying software to a place where a JIT-only VM can’t go, you’re stuck maintaining two VMs in your host program: one you’re using for debug purposes and one that actually ships to the platform you’re interested in. I expect that with time more VMs that include debug protocols and more debuggers will appear. Maybe they will use simpler protocols that don’t require gdb server to be present on the target platform in order to remotely debug and inspect some code, and just require the VM to be built with a debugger-enabled compile option.</p>
<h2 id="binary-size">Binary size</h2>
<p>One of the neat tricks you can do with WebAssembly is update your programs without requiring any native code to be re-deployed on the devices you’re using this on. Other uses involve some sort of compute-at-edge scenarios like Fastly, Cloudflare and others are doing. This is better explained by this <a href="https://www.youtube.com/watch?v=vqBtoPJoQOE">video</a>.</p>
<p>In all of the cases above, or places where disk space is a concern, binary size is one dimension we will need to care about when using WebAssembly. This is rough edge number 2.</p>
<p>I do not want this section to turn into a <em>“Here are 10 tips and tricks to optimize software”</em> kind of blog post. <strong>The most important thing you can do in order to keep this under control is to monitor the size of your WebAssembly binaries from the start of your project</strong>. In doing so, you can find regressions as they happen (e.g. in a CI step) and then proceed to further investigate looking for a fix once your binary size goes above a certain limit.</p>
<p>One advice that I will mention, as it’s generally applicable, is configuring your project to optimize for size and enable LTO. This is done by editing <code class="language-plaintext highlighter-rouge">Cargo.toml</code> to include:</p>
<div class="language-toml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nn">[profile.release]</span>
<span class="py">lto</span> <span class="p">=</span> <span class="kc">true</span>
<span class="py">opt-level</span> <span class="p">=</span> <span class="s">'s'</span> <span class="c"># or 'z', but may cost performance</span>
</code></pre></div></div>
<p>Beyond that, here are some tools that will help you push size optimizations further if needed:</p>
<ol>
<li><a href="https://github.com/WebAssembly/binaryen">wasm-opt</a> can provide further optimizations (beyond the <code class="language-plaintext highlighter-rouge">opt-level='s'</code> flag). You almost always want to use this.</li>
<li><a href="https://github.com/rustwasm/wasm-snip">wasm-snip</a> can be used to remove certain functionality that may be unused. I don’t find it particularly useful as the improvements are not that great (this is it’s a very blunt tool that replaces some methods with <code class="language-plaintext highlighter-rouge">unreachable!</code>). It can be helpful for some use-cases.</li>
<li><a href="https://rustwasm.github.io/twiggy/index.html">twiggy</a> is an excellent tool for finding the top functions that make up for the size of your executable.</li>
<li>Most articles on WebAssembly size optimization will suggest replacing the default rust allocator with <a href="https://github.com/rustwasm/wee_alloc">wee_alloc</a>.</li>
</ol>
<p>Articles that may be helpful:</p>
<ol>
<li><a href="https://rustwasm.github.io/book/reference/code-size.html#optimizing-builds-for-code-size">https://rustwasm.github.io/book/reference/code-size.html#optimizing-builds-for-code-size</a></li>
<li><a href="http://cliffle.com/blog/bare-metal-wasm/">http://cliffle.com/blog/bare-metal-wasm/</a></li>
</ol>Alexandru EneIn part 1 we have learned how to set up WebAssembly VM to run a simple rust program that can add two numbers and print the result to stdout. In part 2 we will go over Debugging and Binary size.Webassembly Without The Browser Part 12020-08-17T00:00:00+00:002020-08-17T00:00:00+00:00http://alexene.dev/2020/08/17/webassembly-without-the-browser-part-1<p>Most WebAssembly tutorials and examples you will find online focus on using it inside the browser in order to accelerate various functionality of a website or web app.<br />
However, there is an area where WebAssembly is really powerful but not talked too much about: outside the browser usage scenarios. That is what we’ll focus on in this series of posts.</p>
<h2 id="what-is-webassembly">What is WebAssembly?</h2>
<p>Web people are on a roll of giving bad names to things (web-gpu is another example).<br />
WebAssembly is neither web or assembly, but a bytecode that can be targeted from languages like C++, C#, Rust and others. This means you can write some Rust code, compile it into WebAssembly and run that code in a WebAssembly virtual machine.</p>
<p>This is powerful because you won’t have to deal with garbage collected scripted languages anymore, and essentially use Rust or C++ as your <em>scripting language</em>. WebAssembly enables predictable and stable performance because it doesn’t require garbage collection like the usual options (LUA/JavaScript).</p>
<p>It’s a relatively new product and there are a lot of rough edges, especially for out-of-browser scenarios. One of the roughest ones in my experience has been documentation for out-of-browser scenarios and this is the reason for my blog posts, to document my findings and hopefully help some people that may be interested in this subject.</p>
<h2 id="why-would-we-want-to-run-webassembly-outside-of-a-browser">Why would we want to run WebAssembly outside of a browser?</h2>
<p>For out of browser scenarios, one of its main advantage is that it provides system level access without compromising on security. This is done through WASI, the Web Assembly System Interface. <a href="https://wasi.dev/">WASI</a> is a collection of C-like functions that provide access to functionality such as <code class="language-plaintext highlighter-rouge">fd_read</code>, <code class="language-plaintext highlighter-rouge">rand</code>, <code class="language-plaintext highlighter-rouge">fd_write</code>, threads (WIP), in a safe way.</p>
<p>Here are a few scenarios where you would be able to use web-assembly outside of a browser:</p>
<ul>
<li>A scripting language for a video game.</li>
<li>To run some code with minimal overhead as Fastly/Cloudflare are doing with their compute-at-edge scenarios.</li>
<li>To run some easy to update code on IoT devices safely and with minimal runtime overhead.</li>
<li>Extreamly fast programs in environments where you can’t JIT for <em>reasons</em>.</li>
</ul>
<h2 id="prerequisites">Prerequisites</h2>
<p>For the best experience in this adventure, I suggest using <a href="https://code.visualstudio.com/">Visual Studio Code</a> as your IDE and install the following extensions:</p>
<ul>
<li><code class="language-plaintext highlighter-rouge">rust-analyzer</code>: for autocomplete and other great features.</li>
<li><code class="language-plaintext highlighter-rouge">Code-LLDB</code>: For debugging with LLDB (even works on Windows)</li>
<li><code class="language-plaintext highlighter-rouge">WebAssembly by the WebAssembly foundation</code>: Allows you to disassemble and inspect <code class="language-plaintext highlighter-rouge">.wasm</code> binaries.</li>
</ul>
<h1 id="choosing-a-virtual-machine">Choosing a Virtual Machine</h1>
<p>First you need a Virtual Machine (VM) that can run your WebAssembly program. This VM needs to be embeddable, so you can add it in your game engine, or what we will call from now on <strong>host program</strong>. There are a few to pick from: <a href="https://github.com/wasm3/wasm3">WASM3</a>, <a href="https://github.com/bytecodealliance/wasmtime">Wasmtime</a>, <a href="https://github.com/bytecodealliance/wasm-micro-runtime">WAMR</a>, and many others. They have various characteristics, such as supporting JIT, using as little memory as possible and so on and you have to choose one one that fits your target platform and scenario.</p>
<p>It doesn’t matter too much what VM you’re choosing besides runtime properties, with the exception of debugging. The only VM that allows for a seamless debugging experience that I’ve found is Wasmtime (this is another one of those rough edges). So even if you don’t plan on deploying that anywhere due to other constraints, I suggest using it as the <strong>debug VM</strong>. Whenever you’d want to debug some WASM code you can launch it with Wasmtime.</p>
<h1 id="writing-our-first-webassembly-program">Writing our first WebAssembly program</h1>
<p>First, we need to create a new <code class="language-plaintext highlighter-rouge">lib</code> project:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>cargo new <span class="nt">--lib</span> wasm_example
</code></pre></div></div>
<p>In <code class="language-plaintext highlighter-rouge">Cargo.toml</code> add the following:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>[lib]
crate-type = ["cdylib"]
</code></pre></div></div>
<p>Now we can edit <code class="language-plaintext highlighter-rouge">lib.rs</code> and export the following <code class="language-plaintext highlighter-rouge">C</code> FFI compatible function from it:</p>
<div class="language-rust highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nd">#[no_mangle]</span>
<span class="k">extern</span> <span class="s">"C"</span> <span class="k">fn</span> <span class="nf">sum</span><span class="p">(</span><span class="n">a</span><span class="p">:</span> <span class="nb">i32</span><span class="p">,</span> <span class="n">b</span><span class="p">:</span> <span class="nb">i32</span><span class="p">)</span> <span class="k">-></span> <span class="nb">i32</span> <span class="p">{</span>
<span class="k">let</span> <span class="n">s</span> <span class="o">=</span> <span class="n">a</span> <span class="o">+</span> <span class="n">b</span><span class="p">;</span>
<span class="nd">println!</span><span class="p">(</span><span class="s">"From WASM: Sum is: {:?}"</span><span class="p">,</span> <span class="n">s</span><span class="p">);</span>
<span class="n">s</span>
<span class="p">}</span>
</code></pre></div></div>
<p>This a function that takes two numbers, adds them, then prints the result before returning their sum.<br />
WebAssembly doesn’t define a default function that’s executed after a module is loaded, so in the host program you need to get a function by it’s signature, and run it (quite similar to how <code class="language-plaintext highlighter-rouge">dlopen</code>/<code class="language-plaintext highlighter-rouge">dlsym</code> works).</p>
<p>We expose this <code class="language-plaintext highlighter-rouge">sum</code> function (and any other functions we want to call from the host VM) as a function that’s callable from <code class="language-plaintext highlighter-rouge">C</code>, using <code class="language-plaintext highlighter-rouge">[#no_mangle]</code> and <code class="language-plaintext highlighter-rouge">pub extern "C"</code>. If you’re coming here from some WASM for the browser tutorials, you may notice we don’t need to use <code class="language-plaintext highlighter-rouge">wasm-bindgen</code> at all.</p>
<h2 id="how-do-we-compile-it">How do we compile it?</h2>
<p>Rust supports two targets for WebAssembly: <code class="language-plaintext highlighter-rouge">wasm32-unknown-unknown</code> and <code class="language-plaintext highlighter-rouge">wasm32-wasi</code>. The first one is bare-bones WebAssembly. Think of it like the <code class="language-plaintext highlighter-rouge">[#no-std]</code> of WebAssembly. It’s the kind you’d use for the browser that doesn’t assume any system functions are available.</p>
<p>At the other end, <code class="language-plaintext highlighter-rouge">wasm32-wasi</code> assumes that the VM exposes the <code class="language-plaintext highlighter-rouge">WASI</code> functionality, allowing a different implementation of the standard library to be used (the implementation that depends on the WASI functions to be available).</p>
<p>You can take a look at the available implementations for the Rust’s stdlib here: <a href="https://github.com/rust-lang/rust/tree/master/library/std/src/sys">https://github.com/rust-lang/rust/tree/master/library/std/src/sys</a> <br />
This is the implementation that assumes WASI functions are available to the rust program when running in a WebAssembly VM: <a href="https://github.com/rust-lang/rust/tree/master/library/std/src/sys/wasi">https://github.com/rust-lang/rust/tree/master/library/std/src/sys/wasi</a>.</p>
<p>To comile for wasm32-wasi run:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c"># Run this just once</span>
rustup target add wasm32-wasi
<span class="c"># Compile for the wasm32-wasi target.</span>
cargo build <span class="nt">--target</span> wasm32-wasi
</code></pre></div></div>
<h2 id="but-how-does-println-work">But how does <code class="language-plaintext highlighter-rouge">println!()</code> work?</h2>
<p>You may have noticed that we’re calling <code class="language-plaintext highlighter-rouge">println!()</code> and expecting the program to work and print to the console, but how does a WebAssembly program knows how to do that?</p>
<p>This is why we’re using <code class="language-plaintext highlighter-rouge">wasm32-wasi</code>. This target selects for the rust stdlib the version that assumes some functionality to be there (the <code class="language-plaintext highlighter-rouge">WASI</code> functions). Printing to the console means just writing to a special file descriptor. Most VMs allow that by default so we don’t need to do any special settings, besides compiling the correct <code class="language-plaintext highlighter-rouge">wasm32-wasi</code> target.</p>
<p>If you have installed the required extensions for vscode, you can now right click on <code class="language-plaintext highlighter-rouge">target/wasm32-wasi/debug/wasm_example.wasm</code> and select <code class="language-plaintext highlighter-rouge">Show WebAssembly</code> and you should have a new file open in vscode that looks like this:</p>
<div class="language-s highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="p">(</span><span class="n">module</span><span class="w">
</span><span class="n">....</span><span class="w">
</span><span class="p">(</span><span class="n">type</span><span class="w"> </span><span class="o">$</span><span class="n">t15</span><span class="w"> </span><span class="p">(</span><span class="n">func</span><span class="w"> </span><span class="p">(</span><span class="n">param</span><span class="w"> </span><span class="n">i64</span><span class="w"> </span><span class="n">i32</span><span class="w"> </span><span class="n">i32</span><span class="p">)</span><span class="w"> </span><span class="p">(</span><span class="n">result</span><span class="w"> </span><span class="n">i32</span><span class="p">)))</span><span class="w">
</span><span class="p">(</span><span class="n">import</span><span class="w"> </span><span class="s2">"wasi_snapshot_preview1"</span><span class="w"> </span><span class="s2">"fd_write"</span><span class="w"> </span><span class="p">(</span><span class="n">func</span><span class="w"> </span><span class="o">$</span><span class="err">_</span><span class="n">ZN4wasi13lib_generated22wasi_snapshot_preview18fd_write17h6ec13d25aa9fb6acE</span><span class="w"> </span><span class="p">(</span><span class="n">type</span><span class="w"> </span><span class="o">$</span><span class="n">t8</span><span class="p">)))</span><span class="w">
</span><span class="p">(</span><span class="n">import</span><span class="w"> </span><span class="s2">"wasi_snapshot_preview1"</span><span class="w"> </span><span class="s2">"proc_exit"</span><span class="w"> </span><span class="p">(</span><span class="n">func</span><span class="w"> </span><span class="o">$</span><span class="err">__</span><span class="n">wasi_proc_exit</span><span class="w"> </span><span class="p">(</span><span class="n">type</span><span class="w"> </span><span class="o">$</span><span class="n">t0</span><span class="p">)))</span><span class="w">
</span><span class="p">(</span><span class="n">import</span><span class="w"> </span><span class="s2">"wasi_snapshot_preview1"</span><span class="w"> </span><span class="s2">"environ_sizes_get"</span><span class="w"> </span><span class="p">(</span><span class="n">func</span><span class="w"> </span><span class="o">$</span><span class="err">__</span><span class="n">wasi_environ_sizes_get</span><span class="w"> </span><span class="p">(</span><span class="n">type</span><span class="w"> </span><span class="o">$</span><span class="n">t2</span><span class="p">)))</span><span class="w">
</span><span class="p">(</span><span class="n">import</span><span class="w"> </span><span class="s2">"wasi_snapshot_preview1"</span><span class="w"> </span><span class="s2">"environ_get"</span><span class="w"> </span><span class="p">(</span><span class="n">func</span><span class="w"> </span><span class="o">$</span><span class="err">__</span><span class="n">wasi_environ_get</span><span class="w"> </span><span class="p">(</span><span class="n">type</span><span class="w"> </span><span class="o">$</span><span class="n">t2</span><span class="p">)))</span><span class="w">
</span><span class="p">(</span><span class="n">func</span><span class="w"> </span><span class="o">$</span><span class="err">_</span><span class="n">ZN4core3fmt9Arguments6new_v117hb11611244be67330E</span><span class="w"> </span><span class="p">(</span><span class="n">type</span><span class="w"> </span><span class="o">$</span><span class="n">t9</span><span class="p">)</span><span class="w"> </span><span class="p">(</span><span class="n">param</span><span class="w"> </span><span class="o">$</span><span class="n">p0</span><span class="w"> </span><span class="n">i32</span><span class="p">)</span><span class="w"> </span><span class="p">(</span><span class="n">param</span><span class="w"> </span><span class="o">$</span><span class="n">p1</span><span class="w"> </span><span class="n">i32</span><span class="p">)</span><span class="w"> </span><span class="p">(</span><span class="n">param</span><span class="w"> </span><span class="o">$</span><span class="n">p2</span><span class="w"> </span><span class="n">i32</span><span class="p">)</span><span class="w"> </span><span class="p">(</span><span class="n">param</span><span class="w"> </span><span class="o">$</span><span class="n">p3</span><span class="w"> </span><span class="n">i32</span><span class="p">)</span><span class="w"> </span><span class="p">(</span><span class="n">param</span><span class="w"> </span><span class="o">$</span><span class="n">p4</span><span class="w"> </span><span class="n">i32</span><span class="p">)</span><span class="w">
</span><span class="p">(</span><span class="n">local</span><span class="w"> </span><span class="o">$</span><span class="n">l5</span><span class="w"> </span><span class="n">i32</span><span class="p">)</span><span class="w"> </span><span class="p">(</span><span class="n">local</span><span class="w"> </span><span class="o">$</span><span class="n">l6</span><span class="w"> </span><span class="n">i32</span><span class="p">)</span><span class="w"> </span><span class="p">(</span><span class="n">local</span><span class="w"> </span><span class="o">$</span><span class="n">l7</span><span class="w"> </span><span class="n">i32</span><span class="p">)</span><span class="w"> </span><span class="p">(</span><span class="n">local</span><span class="w"> </span><span class="o">$</span><span class="n">l8</span><span class="w"> </span><span class="n">i32</span><span class="p">)</span><span class="w"> </span><span class="p">(</span><span class="n">local</span><span class="w"> </span><span class="o">$</span><span class="n">l9</span><span class="w"> </span><span class="n">i32</span><span class="p">)</span><span class="w"> </span><span class="p">(</span><span class="n">local</span><span class="w"> </span><span class="o">$</span><span class="n">l10</span><span class="w"> </span><span class="n">i32</span><span class="p">)</span><span class="w">
</span><span class="n">global.get</span><span class="w"> </span><span class="o">$</span><span class="n">g0</span><span class="w">
</span><span class="n">local.set</span><span class="w"> </span><span class="o">$</span><span class="n">l5</span><span class="w">
</span><span class="n">...</span><span class="w">
</span></code></pre></div></div>
<p>This is a <code class="language-plaintext highlighter-rouge">wat</code> file. <code class="language-plaintext highlighter-rouge">wat</code> stands for WebAssembly text format. It’s kind of like looking at x64/ARM ASM instructions when disassembling a binary, just uglier and harder to understand. I have read that this was because the creators of WebAssembly couldn’t decide on a text format so they just left it in this ugly s-expression form.</p>
<p>The import statements here tell us that the WASM program needs the following functions <code class="language-plaintext highlighter-rouge">proc_exit</code>, <code class="language-plaintext highlighter-rouge">fd_write</code>, <code class="language-plaintext highlighter-rouge">environ_get</code>, <code class="language-plaintext highlighter-rouge">environ_sizes_get</code> to exist in the <code class="language-plaintext highlighter-rouge">wasi_snapshot_preview1</code> namespace.<br />
All imported or exported functions from a WebAssembly module require a namespace. <code class="language-plaintext highlighter-rouge">wasi_snapshot_preview1</code> is the WASI namespace so you can think of it as a reserved namespace for these functions. <code class="language-plaintext highlighter-rouge">println!</code> needs <code class="language-plaintext highlighter-rouge">wasi_snapshot_preview1::fd_write</code> to write to stdout.</p>
<h2 id="the-host-program">The host program</h2>
<p>You can pick any VM that has WASI available. I will use Wasmtime because later on I want to show you how to debug WebAssembly and this VM is the only one where debugging works at the moment.</p>
<p>The program loads the wasm binary file from the path: <code class="language-plaintext highlighter-rouge">examples/wasm_example.wasm</code>.<br />
This is the file you have previously compiled that you can find in <code class="language-plaintext highlighter-rouge">wasm_example/target/wasm32-wasi/debug/wasm_example.wasm</code>. <strong>Make sure you move it in the right place before running the host program.</strong></p>
<p>Here is the full listing of the host VM rust program that initializes the Wasmtime VM, loads the module, links against WASI and loads and executes the exported <code class="language-plaintext highlighter-rouge">sum</code> function from the WASM module:</p>
<div class="language-rust highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">use</span> <span class="nn">std</span><span class="p">::</span><span class="nn">error</span><span class="p">::</span><span class="n">Error</span><span class="p">;</span>
<span class="k">use</span> <span class="nn">wasmtime</span><span class="p">::</span><span class="o">*</span><span class="p">;</span>
<span class="k">use</span> <span class="nn">wasmtime_wasi</span><span class="p">::{</span><span class="n">Wasi</span><span class="p">,</span> <span class="n">WasiCtx</span><span class="p">};</span>
<span class="k">fn</span> <span class="nf">main</span><span class="p">()</span> <span class="k">-></span> <span class="n">Result</span><span class="o"><</span><span class="p">(),</span> <span class="nb">Box</span><span class="o"><</span><span class="n">dyn</span> <span class="n">Error</span><span class="o">>></span> <span class="p">{</span>
<span class="c">// A `Store` is a sort of "global object" in a sense, but for now it suffices</span>
<span class="c">// to say that it's generally passed to most constructors.</span>
<span class="c">// let store = Store::default();</span>
<span class="k">let</span> <span class="n">engine</span> <span class="o">=</span> <span class="nn">Engine</span><span class="p">::</span><span class="nf">new</span><span class="p">(</span><span class="nn">Config</span><span class="p">::</span><span class="nf">new</span><span class="p">()</span><span class="nf">.debug_info</span><span class="p">(</span><span class="k">true</span><span class="p">));</span>
<span class="k">let</span> <span class="n">store</span> <span class="o">=</span> <span class="nn">Store</span><span class="p">::</span><span class="nf">new</span><span class="p">(</span><span class="o">&</span><span class="n">engine</span><span class="p">);</span>
<span class="c">// We start off by creating a `Module` which represents a compiled form</span>
<span class="c">// of our input wasm module. In this case it'll be JIT-compiled after</span>
<span class="c">// we parse the text format.</span>
<span class="k">let</span> <span class="n">module</span> <span class="o">=</span> <span class="nn">Module</span><span class="p">::</span><span class="nf">from_file</span><span class="p">(</span><span class="o">&</span><span class="n">engine</span><span class="p">,</span> <span class="s">"examples/wasm_example.wasm"</span><span class="p">)</span><span class="o">?</span><span class="p">;</span>
<span class="c">// Link the WASI module to our VM. Wasmtime allows us to decide if WASI is present.</span>
<span class="c">// So we need to load it here, as our module rquires certain functions to be present from the</span>
<span class="c">// wasi_snapshot_preview1 namespace as seen above.</span>
<span class="c">// This makes println!() from our WASM program to work. (it uses fd_write).</span>
<span class="k">let</span> <span class="n">wasi</span> <span class="o">=</span> <span class="nn">Wasi</span><span class="p">::</span><span class="nf">new</span><span class="p">(</span><span class="o">&</span><span class="n">store</span><span class="p">,</span> <span class="nn">WasiCtx</span><span class="p">::</span><span class="nf">new</span><span class="p">(</span><span class="nn">std</span><span class="p">::</span><span class="nn">env</span><span class="p">::</span><span class="nf">args</span><span class="p">())</span><span class="o">?</span><span class="p">);</span>
<span class="k">let</span> <span class="k">mut</span> <span class="n">imports</span> <span class="o">=</span> <span class="nn">Vec</span><span class="p">::</span><span class="nf">new</span><span class="p">();</span>
<span class="k">for</span> <span class="n">import</span> <span class="n">in</span> <span class="n">module</span><span class="nf">.imports</span><span class="p">()</span> <span class="p">{</span>
<span class="k">if</span> <span class="n">import</span><span class="nf">.module</span><span class="p">()</span> <span class="o">==</span> <span class="s">"wasi_snapshot_preview1"</span> <span class="p">{</span>
<span class="k">if</span> <span class="k">let</span> <span class="nf">Some</span><span class="p">(</span><span class="n">export</span><span class="p">)</span> <span class="o">=</span> <span class="n">wasi</span><span class="nf">.get_export</span><span class="p">(</span><span class="n">import</span><span class="nf">.name</span><span class="p">())</span> <span class="p">{</span>
<span class="n">imports</span><span class="nf">.push</span><span class="p">(</span><span class="nn">Extern</span><span class="p">::</span><span class="nf">from</span><span class="p">(</span><span class="n">export</span><span class="nf">.clone</span><span class="p">()));</span>
<span class="k">continue</span><span class="p">;</span>
<span class="p">}</span>
<span class="p">}</span>
<span class="nd">panic!</span><span class="p">(</span>
<span class="s">"couldn't find import for `{}::{}`"</span><span class="p">,</span>
<span class="n">import</span><span class="nf">.module</span><span class="p">(),</span>
<span class="n">import</span><span class="nf">.name</span><span class="p">()</span>
<span class="p">);</span>
<span class="p">}</span>
<span class="c">// After we have a compiled `Module` we can then instantiate it, creating</span>
<span class="c">// an `Instance` which we can actually poke at functions on.</span>
<span class="k">let</span> <span class="n">instance</span> <span class="o">=</span> <span class="nn">Instance</span><span class="p">::</span><span class="nf">new</span><span class="p">(</span><span class="o">&</span><span class="n">store</span><span class="p">,</span> <span class="o">&</span><span class="n">module</span><span class="p">,</span> <span class="o">&</span><span class="n">imports</span><span class="p">)</span><span class="o">?</span><span class="p">;</span>
<span class="c">// The `Instance` gives us access to various exported functions and items,</span>
<span class="c">// which we access here to pull out our `answer` exported function and</span>
<span class="c">// run it.</span>
<span class="k">let</span> <span class="n">main</span> <span class="o">=</span> <span class="n">instance</span><span class="nf">.get_func</span><span class="p">(</span><span class="s">"sum"</span><span class="p">)</span>
<span class="nf">.expect</span><span class="p">(</span><span class="s">"`main` was not an exported function"</span><span class="p">);</span>
<span class="c">// There's a few ways we can call the `main` `Func` value. The easiest</span>
<span class="c">// is to statically assert its signature with `get2` (in this case asserting</span>
<span class="c">// it takes 2 i32 arguments and returns one i32) and then call it.</span>
<span class="k">let</span> <span class="n">main</span> <span class="o">=</span> <span class="n">main</span><span class="py">.get2</span><span class="p">::</span><span class="o"><</span><span class="nb">i32</span><span class="p">,</span> <span class="nb">i32</span><span class="p">,</span> <span class="nb">i32</span><span class="o">></span><span class="p">()</span><span class="o">?</span><span class="p">;</span>
<span class="c">// And finally we can call our function! Note that the error propagation</span>
<span class="c">// with `?` is done to handle the case where the wasm function traps.</span>
<span class="k">let</span> <span class="n">result</span> <span class="o">=</span> <span class="nf">main</span><span class="p">(</span><span class="mi">5</span><span class="p">,</span> <span class="mi">4</span><span class="p">)</span><span class="o">?</span><span class="p">;</span>
<span class="nd">println!</span><span class="p">(</span><span class="s">"From host: Answer returned to the host VM: {:?}"</span><span class="p">,</span> <span class="n">result</span><span class="p">);</span>
<span class="nf">Ok</span><span class="p">(())</span>
<span class="p">}</span>
</code></pre></div></div>
<p>The <code class="language-plaintext highlighter-rouge">Cargo.toml</code> of this project needs to have the following dependencies:</p>
<div class="language-toml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nn">[dependencies]</span>
<span class="py">wasmtime</span> <span class="p">=</span> <span class="s">"0.19"</span>
<span class="py">wasmtime-wasi</span> <span class="p">=</span> <span class="s">"0.19"</span>
<span class="py">anyhow</span> <span class="p">=</span> <span class="s">"1.0.28"</span>
</code></pre></div></div>
<p>Running this with <code class="language-plaintext highlighter-rouge">cargo run</code> will print the following output:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>Compiling wasm_host v0.1.0 (wasm_host)
Finished dev [unoptimized + debuginfo] target(s) in 35.38s
Running `target\debug\wasm_host.exe`
From WASM: Sum is: 9
From host: Answer returned to the host VM: 9
</code></pre></div></div>
<p>We can observe that the <code class="language-plaintext highlighter-rouge">println!</code> from the wasm module has correctly printed to the console and that the returned answer is as expected <code class="language-plaintext highlighter-rouge">9</code>.</p>
<h2 id="conclusion">Conclusion</h2>
<p>In this post of my WebAssembly Outside the Browser series we’ve learned how to compile a program for WebAssembly, set-up a host program to load and run a your WASM binary, execute a function exported by the WASM program and put all that together we ended up adding two numbers and printing their result (from both WebAssembly and the host program).<br />
In the next parts we will touch areas such as debugging, optimizing program size, exposing functions from the host vm to the WASM program and sharing memory between the two VMs.</p>
<h2 id="bonus-study-materials">Bonus study materials</h2>
<p>Here is the full <a href="https://webassembly.github.io/spec/core/index.html">WASM specification</a>. For me that it’s one of the hardest spec that I’ve ever had to read.<br />
I would much rather have this spec similar to a CPU user manual (e.g. <a href="http://datasheets.chipdb.org/NEC/Vr-Series/Vr43xx/U10504EJ7V0UMJ1.pdf">VR4300</a>), rather than it’s current form that is forced into some math-y language that, while correct, brings no extra clarity or insight to the reader.<br />
I strongly think that the concepts described there could have been very well expressed in an easier to understand and parse language and I don’t buy the usual excuse that <em>“Well actually, it’s targeted at VM writers, not normal people”</em>. We should just accept that it’s not accessible at all, and we could do way better.</p>
<p><strong>Some more materials:</strong><br />
Videos:<br />
<a href="https://www.youtube.com/watch?v=vqBtoPJoQOE">Kevin Hoffman: Building a Containerless Future with WebAssembly</a><br />
<a href="https://www.youtube.com/watch?v=C8j_ieOm4vE">Peter Salomonsen: WebAssembly Music</a></p>
<p>Reading material:<br />
<a href="https://webassembly.org/">WebAssembly.org</a><br />
<a href="https://hacks.mozilla.org/2019/03/standardizing-wasi-a-webassembly-system-interface/">Standardizing WASI: A system interface to run WebAssembly outside the web</a><br />
<a href="http://cliffle.com/blog/bare-metal-wasm/">Cliff L. Biffle: Making really tiny WebAssembly graphics demos</a><br />
<a href="https://labs.imaginea.com/talk-the-nuts-and-bolts-of-webassembly/">An overview of WebAssembly’s historical context</a></p>Alexandru EneMost WebAssembly tutorials and examples you will find online focus on using it inside the browser in order to accelerate various functionality of a website or web app. However, there is an area where WebAssembly is really powerful but not talked too much about: outside the browser usage scenarios. That is what we’ll focus on in this series of posts.Physically Based Temperature Simulation For Games2020-01-10T00:00:00+00:002020-01-10T00:00:00+00:00http://alexene.dev/2020/01/10/Physically-based-temperature-simulation-for-games<blockquote>
<p>I wish today’s games would have accurate temperature simulation…</p>
</blockquote>
<p>I hear that all the time. <br />
We already have physically-based rendering, so why did we stop there?!</p>
<h2 id="but-why">But… Why?</h2>
<p>Because it’s my game and I do whatever I want.<br />
I am working on a sim game where you have to take care of a colony of dwarves and I want seasons to play a big part in this.<br />
I want my dwarves to be cold in the winter, get warm in the summer, lose a limb due to frostbites. Many games have already experimented with such systems to some degree (e.g. Don’t Starve).</p>
<p><strong>But can’t that be solved with a simple radius check and a decay formula for the temperature transferred?</strong> <br />
Yes you can and you should do that, but for me that wasn’t enough for a few reasons:<br />
1) I wanted to have cave temperatures be slowly influenced by the outside temperatures. E.g. you can grow mushrooms in your cave but not near the cave entrance as they might need a cooler and more stable temperature.<br />
2) I want certain plants and foods to preserve better at lower temperatures.<br />
3) Another thing that complicated my life are walls. If you have a fire in one room near a wall you don’t heat the air outside of that room at the same rate as you heat the room.</p>
<h2 id="the-solution">The solution</h2>
<p>In addition to the reasons mentioned above it is really hard to tune random formulas that have nothing to do with reality. It’s way easier to start from a realistic system and formulas and tune that to make it fun.</p>
<p>Let’s go to the classic formula of calculating the heat transferred when a system goes from one temperature to another.</p>
<div>
$$
Q = m * c * {\Delta}T
$$
</div>
<p><code class="language-plaintext highlighter-rouge">Q</code> = Heat, measured in Joules - <code class="language-plaintext highlighter-rouge">J</code> <br />
<code class="language-plaintext highlighter-rouge">m</code> = mass of the system grams - <code class="language-plaintext highlighter-rouge">g</code><br />
<code class="language-plaintext highlighter-rouge">c</code> = specific heat capacity of a substance. That is the energy required (heat) such that a unit of mass (g) of that substance will raise its temperature by one unit (Kelvins - <code class="language-plaintext highlighter-rouge">K</code>).</p>
<p>So, as far as units of measurement are, we have an equation of the form:</p>
<div>
$$
J = g * \dfrac{J}{g*K} * K
$$
</div>
<p>So it all checks out and we are left with <code class="language-plaintext highlighter-rouge">Q</code> being measured in Joules.</p>
<p>But just that formula is not enough as it doesn’t tell us how two adjacent terrain cells at different temperatures should transfer heat between each other.<br />
When two bodies have different temperatures, their molecules have different average kinetic energies (they bounce around at different speeds). When these two bodies are in contact, collisions between moving molecules on the surface of contact will transfer energy from the high-temperature body to the low-temperature one.<br />
As that transfer happens, the higher temperature body slightly cools down and the lower temperature body warms up.<br />
We intuitively already know that different materials conduct energy at different rates. That’s why you have air in between two sheets of glass in your windows. Air is a material that has a high thermal resistivity.</p>
<p>If you research this online you find these material constants defined as either <code class="language-plaintext highlighter-rouge">thermal resistivity</code> or <code class="language-plaintext highlighter-rouge">thermal conductivity</code>. They are the basically expressing the same thing, but you divide by one and multiply by the other.</p>
<p>For thermal conduction we have this formula using conductivity:</p>
<div>
$$
\dfrac{Q}{t} = \dfrac{(k * A * {\Delta}T)}{d}
$$
</div>
<p>Where:<br />
<code class="language-plaintext highlighter-rouge">t</code> = time <br />
<code class="language-plaintext highlighter-rouge">Q</code> = heat<br />
<code class="language-plaintext highlighter-rouge">A</code> = surface area<br />
<code class="language-plaintext highlighter-rouge">T</code> = temperature<br />
<code class="language-plaintext highlighter-rouge">d</code> = distance to the point we want to measure the temperature at.<br />
<code class="language-plaintext highlighter-rouge">k</code> = thermal conductivity</p>
<p>If we use thermal resistivity it looks like this:</p>
<div>
$$
\dfrac{Q}{t} = \dfrac{A*{\Delta}T}{resistivity * d}
$$
</div>
<p>I went with the thermal resistivity version as engineering books seem to favor it and they provide constants for a variety of building materials I have in my game. If you find conductivity values, you can easily switch from one to the other. <strong>Just make sure your units of measurement work out correctly. If you express everything in SI units you will be fine.</strong></p>
<h2 id="putting-it-all-together">Putting it all together</h2>
<p>The terrain in my game is a 3-D grid of cells (think minecraft-like). Some cells are made of ground, some are air, stone, etc.<br />
That being said, update cycle that happens for temperatures in my game is made up of the following steps.</p>
<h3 id="0---heat-propagation-from-the-sun">0 - Heat propagation from the “sun”</h3>
<p>All terrain cells that <em>see</em> the sky will get their temperatures updated based on some ambient temperature based on the time of day and season.
All the other terrain cells will tend to get towards a different ambient temperature based on how deep they are, etc.
This step is here to simulate a fake convection and radiation. Without it all the map will eventually heat up from a fireplace (as there would be just energy added, and none lost).<br />
It’s also very easy to tune seasons temperatures like this so that’s why nothing complex is going on this step.</p>
<h3 id="1---heat-propagation-for-environment-terrain-cells">1 - Heat propagation for environment (terrain cells)</h3>
<p>For this we apply this formula as we need to know <code class="language-plaintext highlighter-rouge">Q</code>, the heat transferred between two adjacent terrain cells.</p>
<div>
$$
{Q} = \dfrac{A*t*{\Delta}T}{resistivity * d}
$$
</div>
<p>Using the formula above we compute the average energy a terrain cell is transferring with its neighbors.
I consider a cell’s temperature as the temperature measured at the center of that cell.
This energy is used to compute a cell’s new temperature by plugging <code class="language-plaintext highlighter-rouge">Q</code> in this equation:</p>
<div>
$$
T_{Final} = T_{Initial} + \dfrac{Q}{m*c}
$$
</div>
<p>We have the final temperature of our terrain cell.<br />
You can see how this process is super easy to parallelize, as each cell updates it’s temperature from the previous temperature of the neighbor cells. You just need to have two terrain copies (one with the old temperatures, and one that will get the new temperatures based on the old values).</p>
<h3 id="2---heat-propagation-to-game-entities">2 - Heat propagation to game entities</h3>
<p>For this we use the same formulas as above, with a small exception. We just update the game entity’s temperature and we don’t change the terrain cell temperature based on that game entity. This is for <em>receiving</em> game units (dwarves, objects, cows, etc.). These entities have a MaterialComponent attached to them. That component tracks the temperature and other useful material properties to compute that temperature. Applying the proper formula where the dwarf also affects the game cell temperature doesn’t yeld nice results due to the unrealistic size ratio between a terrain cell and a dwarf (see bottom of the article).<br />
<code class="language-plaintext highlighter-rouge">Emitting</code> game units like campfires propagate the temperature in the other direction, from them to the terrain.</p>
<p>This is the definition of the <code class="language-plaintext highlighter-rouge">MaterialComponent</code>:</p>
<div class="language-rust highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nd">#[derive(Debug,</span> <span class="nd">Serialize,</span> <span class="nd">Deserialize)]</span>
<span class="k">pub</span> <span class="k">struct</span> <span class="n">MaterialComponent</span> <span class="p">{</span>
<span class="n">volume</span><span class="p">:</span> <span class="nb">f32</span><span class="p">,</span>
<span class="n">area</span><span class="p">:</span> <span class="nb">f32</span><span class="p">,</span>
<span class="n">temperature</span><span class="p">:</span> <span class="nb">f32</span><span class="p">,</span>
<span class="n">radius</span><span class="p">:</span> <span class="nb">f32</span><span class="p">,</span>
<span class="n">mass</span><span class="p">:</span> <span class="nb">f32</span><span class="p">,</span>
<span class="n">material</span><span class="p">:</span> <span class="n">Material</span><span class="p">,</span>
<span class="p">}</span>
</code></pre></div></div>
<p>Let’s remember the properties needed to transfer temperature between two entities:</p>
<ul>
<li>Surface area</li>
<li>Initial_temperature</li>
<li>Distance (that the energy <em>travels</em>)</li>
<li>Specific specific heat capacity</li>
<li>Thermal resistivity</li>
<li>Time</li>
<li>Mass</li>
</ul>
<p>From that list, just a few are entity-specific: <strong>surface area</strong>, <strong>distance</strong>, <strong>mass</strong>.<br />
Besides time, the rest depend on the material so we can keep them in a separate <code class="language-plaintext highlighter-rouge">material_properties.json</code> table.<br />
That is a json to make it easy to add and tune materials.</p>
<p><strong>But how do we know the mass, surface area and distance that the heat travels for a dwarf body part?</strong></p>
<p>Configuring the <strong>distance energy travels</strong>, <strong>mass</strong> and <strong>surface area</strong> for each body part component is an interesting exercise, but I am lazy and here I went with an approximation with the goal of keeping things intuitive and easy to set-up.</p>
<p>The only parameters actually needed to initialize the <code class="language-plaintext highlighter-rouge">MaterialComponent</code> for an entity (head, hand, leg, etc.) are:</p>
<div class="language-json highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="p">{</span><span class="w">
</span><span class="nl">"MaterialComponent"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nl">"mass"</span><span class="p">:</span><span class="w"> </span><span class="mi">6000</span><span class="p">,</span><span class="w">
</span><span class="nl">"material"</span><span class="p">:</span><span class="w"> </span><span class="s2">"Water"</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">}</span><span class="w">
</span></code></pre></div></div>
<p>Besides the temperature-related <code class="language-plaintext highlighter-rouge">material_properties.json</code> entry, I also input the density in that table.
For example the entry for <code class="language-plaintext highlighter-rouge">"Water"</code> is:</p>
<div class="language-json highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nl">"Water"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nl">"density"</span><span class="p">:</span><span class="w"> </span><span class="mf">997000.0</span><span class="p">,</span><span class="w">
</span><span class="nl">"thermal_resistivity"</span><span class="p">:</span><span class="w"> </span><span class="mf">1.6</span><span class="p">,</span><span class="w">
</span><span class="nl">"specific_heat_capacity"</span><span class="p">:</span><span class="w"> </span><span class="mf">4.2</span><span class="w">
</span><span class="p">}</span><span class="w">
</span></code></pre></div></div>
<p>You probably know where this is going.<br />
<strong>For the purpose of temperature transfer, I am approximating all entities to simple shapes (spheres).</strong></p>
<p>That’s where the density comes into play. With it and the mass we can compute the volume of our entity:</p>
<div>
$$
Volume = \dfrac{Mass}{Density}
$$
</div>
<p>Having the <code class="language-plaintext highlighter-rouge">Volume</code>, using a simple formula we can compute the <code class="language-plaintext highlighter-rouge">radius</code> of the sphere and from that the <code class="language-plaintext highlighter-rouge">area</code>.<br />
We now have all the data needed, so we do the same calculations using the same formulas we used for the terrain temperature.<br />
All <code class="language-plaintext highlighter-rouge">MaterialComponents</code> can be updated in parallel.</p>
<h3 id="3-needs-system">3. Needs system</h3>
<p>The needs system reads the temperature values from the <code class="language-plaintext highlighter-rouge">MaterialComponent</code> and issues Actions based on the needs of a creature(dwarf, cow, etc.). E.g. If they are cold they go near the fire. I will write more about the needs system and how AI works in a different blog post.</p>
<h2 id="next-steps">Next steps</h2>
<p>The next steps is to take armor and clothes into account when calculating the heat transferred. That just slightly changes the formula we have as we just need to add the R-values of all materials involved in the heat transfer:</p>
<div>
$$
R_value = thermal_resistivity * distance
Total_R_Value = Rvalue_of_entity_1 + RValue_of_entity_2 + ...
$$
</div>
<p>Using spheres will make it a bit tricky to handle clothes as the clothes would need to be hollow and I will probably consider all entities with <code class="language-plaintext highlighter-rouge">MaterialComponents</code> cube-shaped instead.<br />
I plan on adding more temperature related effects.<br />
Conduction is just one way heat is transferred between things. Another way to exchange heat is convection and radiation. I am not sure if I will add them as the effects in this game world will probably be so small it’s not worth it.</p>
<p>Because you reached the end of this, I leave you with this cozy picture of a dwarf and a cow warming up near a fire.</p>
<p><img src="/images/temperature/dwarf_and_cow.png" alt="dwarf_and_cow_by_the_fire" /></p>Alexandru EneI wish today’s games would have accurate temperature simulation…Github Actions CI with Rust and SDL22019-09-04T00:00:00+00:002019-09-04T00:00:00+00:00http://alexene.dev/2019/09/04/Github-actions-CI-rust-SDL2<p>Github has added CI workflows with <a href="https://github.blog/2019-08-08-github-actions-now-supports-ci-cd/">github actions</a> and I decided to try it after I saw <a href="https://github.com/yak32/glw_json/blob/master/.github/workflows/main.yml">how well it worked</a> for my colleague, Yakov.<br />
I am doing a game in Rust and I need to test it on Windows, Linux and MacOS.<br />
While Windows and Linux tests are easy to run with the help of WSL on my machine, I don’t own a Mac so that platform was never tested until now.</p>
<p>Besides supporting all three platforms I was interested in, Github also <a href="https://github.com/features/actions">offers</a> a good range of pay as you go prices with a cost per minute of: 0.008$ for Linux, 0.016$ for Windows and 0.08$ for MacOS as well as <strong>2000 minutes of a free tier</strong>.<br />
This is great news for hobby projects like mine that happen to be on a private github repository.</p>
<p>Below you can find the action that I’m currently using.<br />
It installs SDL2 on Linux and MacOS and assumes the DLLs for Windows are in the git repo.<br />
If you think that pushing some DLLs in a git repo is not a clean solution, there’s the alternative of using something like <a href="https://github.com/AlexEne/rust-particles/blob/master/appveyor_get_sdl_dll.ps1">this PowerShell script</a> to get the SDL 2 DLLs.</p>
<p>It comes with the stable Rust 1.37 version on Linux and the Windows distribution.
I had to manually install it on MacOS.<br />
Using <code class="language-plaintext highlighter-rouge">rustup</code> you can also get <code class="language-plaintext highlighter-rouge">nightly/beta</code> versions if needed.</p>
<p>More information on how to set up github actions CI for your own projects:</p>
<ul>
<li><a href="https://help.github.com/en/articles/virtual-environments-for-github-actions">Virtual environments available</a></li>
<li><a href="https://help.github.com/en/articles/workflow-syntax-for-github-actions">Workflow yaml syntax</a></li>
</ul>
<h2 id="the-action">The action</h2>
<div class="language-yaml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="na">name</span><span class="pi">:</span> <span class="s">Rust</span>
<span class="na">on</span><span class="pi">:</span> <span class="pi">[</span><span class="nv">push</span><span class="pi">]</span>
<span class="na">jobs</span><span class="pi">:</span>
<span class="na">test_Ubuntu</span><span class="pi">:</span>
<span class="na">runs-on</span><span class="pi">:</span> <span class="s">ubuntu-latest</span>
<span class="na">steps</span><span class="pi">:</span>
<span class="pi">-</span> <span class="na">uses</span><span class="pi">:</span> <span class="s">actions/checkout@v1</span>
<span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">install_dependencies</span>
<span class="na">run</span><span class="pi">:</span> <span class="pi">|</span>
<span class="s">sudo add-apt-repository -y "deb http://archive.ubuntu.com/ubuntu `lsb_release -sc` main universe restricted multiverse"</span>
<span class="s">sudo apt-get update -y -qq</span>
<span class="s">sudo apt-get install libsdl2-dev</span>
<span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">Build</span>
<span class="na">run</span><span class="pi">:</span> <span class="pi">|</span>
<span class="s">rustc --version</span>
<span class="s">cargo build</span>
<span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">Test</span>
<span class="na">run</span><span class="pi">:</span> <span class="s">cargo test</span>
<span class="na">test_MacOS</span><span class="pi">:</span>
<span class="na">runs-on</span><span class="pi">:</span> <span class="s">macOS-latest</span>
<span class="na">steps</span><span class="pi">:</span>
<span class="pi">-</span> <span class="na">uses</span><span class="pi">:</span> <span class="s">actions/checkout@v1</span>
<span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">install_dependencies</span>
<span class="na">run</span><span class="pi">:</span> <span class="pi">|</span>
<span class="s">brew install SDL2</span>
<span class="s">brew install rustup</span>
<span class="s">rustup-init -y --default-toolchain stable </span>
<span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">Build</span>
<span class="na">run</span><span class="pi">:</span> <span class="pi">|</span>
<span class="s">export PATH="$HOME/.cargo/bin:$PATH"</span>
<span class="s">cargo build</span>
<span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">Test</span>
<span class="na">run</span><span class="pi">:</span> <span class="pi">|</span>
<span class="s">export PATH="$HOME/.cargo/bin:$PATH"</span>
<span class="s">cargo test</span>
<span class="na">test_Windows</span><span class="pi">:</span>
<span class="na">runs-on</span><span class="pi">:</span> <span class="s">windows-2016</span>
<span class="na">steps</span><span class="pi">:</span>
<span class="pi">-</span> <span class="na">uses</span><span class="pi">:</span> <span class="s">actions/checkout@v1</span>
<span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">Build</span>
<span class="na">run</span><span class="pi">:</span> <span class="s">cargo build</span>
<span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">Test</span>
<span class="na">run</span><span class="pi">:</span> <span class="s">cargo test</span>
</code></pre></div></div>
<h2 id="how-it-looks-like">How it looks like</h2>
<p><img src="/images/github-actions-ci/github_actions_ci.gif" alt="split map" /></p>
<h2 id="some-details">Some details</h2>
<p>On <code class="language-plaintext highlighter-rouge">ubuntu-latest</code> I had to do some tricks to install <code class="language-plaintext highlighter-rouge">libsdl2-dev</code>.<br />
Just doing <code class="language-plaintext highlighter-rouge">sudo apt-get install libsdl2-dev</code> doesn’t work right now as it has a missing package.</p>
<p>On Windows I use <code class="language-plaintext highlighter-rouge">windows-2016</code> it has Visual Studio 2017. For now, that’s the newest version supported by rust-hawktracer.<br />
After I will update my build script for rust-hawktracer to handle Visual Studio 2019, <code class="language-plaintext highlighter-rouge">windows-latest</code> will work too.</p>
<p>You can also set it up for PRs like this:</p>
<div class="language-yaml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="na">on</span><span class="pi">:</span>
<span class="na">pull_request</span><span class="pi">:</span>
<span class="na">branches</span><span class="pi">:</span>
<span class="pi">-</span> <span class="s">master</span>
</code></pre></div></div>Alexandru EneGithub has added CI workflows with github actions and I decided to try it after I saw how well it worked for my colleague, Yakov. I am doing a game in Rust and I need to test it on Windows, Linux and MacOS. While Windows and Linux tests are easy to run with the help of WSL on my machine, I don’t own a Mac so that platform was never tested until now.Testing Setup2019-07-06T00:00:00+00:002019-07-06T00:00:00+00:00http://alexene.dev/2019/07/06/Testing-setup<p>Usually videogame developers have three possible approaches to testing:</p>
<ul>
<li>Hire armies of people to do it for you.</li>
<li>Early access.</li>
<li>Hope all is fine.</li>
</ul>
<p>We all know how good the support in rust is for writing tests and I would like to show you my improved testing setup for the game I’m working on.</p>
<h1 id="testing">Testing</h1>
<p>I just test the high-level behavior as things evolve quite fast in my game and having super detailed unit tests for all individual parts is not really worth the effort. At the end of the day all I care about is that if I tell a dwarf to dig a whole at a certain position, he digs a whole at that position.<br />
Time is also a key part of this strategy. I just don’t have enough time to have a lot of detailed unit tests so these high-level tests have to do.</p>
<p>In my previous <a href="2019-01-15-After-hours-game-development.md">post</a> I said:<br />
<em>“Almost all tests are instantiating worlds and various entities and scenarios. This means that if a test breaks, I just copy the world initialization code to the main game and I can visualize that test scenario and debug it really easy, pausing the simulation, inspecting entities with the debug UI, etc.”</em> - me</p>
<p>Copy-pasting things got annoying after some big refactors where I had to check why I was failing a bunch of test cases.<br />
That prompted me to change my testing setup so here’s my <strong>improved</strong> approach.</p>
<p>Today my tests look like this:</p>
<div class="language-rust highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nd">#[test]</span>
<span class="k">fn</span> <span class="nf">dwarf_eventually_dies_of_hunger</span><span class="p">()</span> <span class="p">{</span>
<span class="k">let</span> <span class="k">mut</span> <span class="n">world</span> <span class="o">=</span> <span class="nn">World</span><span class="p">::</span><span class="nf">new</span><span class="p">();</span>
<span class="n">world</span><span class="nf">.spawn_from_entity_type</span><span class="p">(</span><span class="s">"Dwarf"</span><span class="p">,</span> <span class="nn">Vec3</span><span class="p">::</span><span class="nf">new</span><span class="p">(</span><span class="mi">0</span><span class="p">,</span> <span class="mi">0</span><span class="p">,</span> <span class="mi">0</span><span class="p">));</span>
<span class="k">let</span> <span class="k">mut</span> <span class="n">game</span> <span class="o">=</span> <span class="nn">Game</span><span class="p">::</span><span class="nf">new</span><span class="p">(</span><span class="n">world</span><span class="p">,</span> <span class="k">false</span><span class="p">);</span>
<span class="n">game</span><span class="nf">.update</span><span class="p">(</span><span class="nf">Some</span><span class="p">(</span><span class="mi">3000</span><span class="p">));</span>
<span class="c">//RIP</span>
<span class="k">let</span> <span class="n">world</span> <span class="o">=</span> <span class="n">game</span><span class="nf">.get_world</span><span class="p">();</span>
<span class="k">assert</span><span class="o">!</span><span class="p">(</span><span class="n">world</span><span class="nf">.get_entity_by_type</span><span class="p">(</span><span class="s">"Dwarf"</span><span class="p">)</span><span class="nf">.is_none</span><span class="p">());</span>
<span class="p">}</span>
</code></pre></div></div>
<p>At a first glance this looks exactly as the old tests that I presented in the other blog post.<br />
However, there’s an important difference, it is using the new <code class="language-plaintext highlighter-rouge">Game</code> struct.<br />
I can enable the full experience: graphics, window, events, UI just by changing <code class="language-plaintext highlighter-rouge">Game::new(world, false)</code> to <code class="language-plaintext highlighter-rouge">Game::new(world, true)</code>.<br />
This means that I can click on objects, inspect, even change tests as they happen by issuing build orders.</p>
<p>And here’s the above test running:</p>
<p><img src="/images/testing/test_hunger.gif" alt="testing" /></p>
<p>I even tried hard to have the graphics always on for all tests but, due to a <a href="(https://github.com/ocornut/imgui/issues/586)">limitation</a> of IMGUI, I can’t spawn more than one instance of it on multiple threads so this is out of the question.</p>
<p>I consider this new setup a big step forward for my productivity in debugging tests and in adding writing new tests as right now it’s super easy to check that a test checks the right thing.</p>Alexandru EneUsually videogame developers have three possible approaches to testing: Hire armies of people to do it for you. Early access. Hope all is fine.Hierarchical Pathfinding2019-06-02T00:00:00+00:002019-06-02T00:00:00+00:00http://alexene.dev/2019/06/02/Hierarchical-pathfinding<p>Pathfinding gave me a lot of problems as I went with the clasic A* approach for my game. This worked well except one case. When you don’t have a path to a certain point, you might end up traversing all the blocks of a world desperately trying to find one. This can be quite time consuming, and even for a relatively small 100x100 world, you might spend <code class="language-plaintext highlighter-rouge">7ms</code> searching for a path that doesn’t exist. <code class="language-plaintext highlighter-rouge">7ms</code> is a huge amount of time from a game frame of <code class="language-plaintext highlighter-rouge">16ms</code> so I had to do something about this.</p>
<h2 id="intro">Intro</h2>
<p>There are simple solutions to this problem, one of the most popular ones, that I actually ended up implementing is HPA* or as the original paper calls it, Near-Optimal Hierarchical Pathfinding. But in reality it’s just Hirerarchical A*.<br />
This is best explained in the paper <a href="https://webdocs.cs.ualberta.ca/~mmueller/ps/hpastar.pdf">here</a>.</p>
<p>However, I did some changes in order to adapt that paper to my use case – a minecraft-like world, with multiple levels. That paper only solves it for the 2D case, but it’s easily extended to a 3D-block world.</p>
<h2 id="hpa">HPA*</h2>
<p>So how does it work?</p>
<p>The main idea of hierarchical A* is to (you guessed it), create a hierarchy first. So we divide our world into bigger cells, and then compute the path and connection points in between these big cells.</p>
<p><img src="/images/hierarchical_pathfinding/split_world.png" alt="split map" title="A map being split into high-level cells" /></p>
<p>As you can see in the image above, the world is split into 10x10 big cells (the yellow cells). The nice thing about this is that you don’t need to tune anything (except maybe the high-level cell sizes). For now let’s keep them at 10x10 and we have this world divided into high-level cells.</p>
<p>Once we did this, we proceed and find the connection points in between high level cells. These are the points on the edges. There can be quite a lot of them in the case of an open field and here the paper goes one step further. It makes a sort of doors between two adjacent high-level cells. This way it can only keep 2 connection points for each door over a certain size. I didn’t do that but it can be a good optimization.</p>
<p>So right now we have a series of connection points for the high level cells. You can see them with the blue in the picture below. We also added some obstacles (black cells).</p>
<p><img src="/images/hierarchical_pathfinding/generated_connection_points.png" alt="connection points" /></p>
<p>Now we add connection points in between these cells. First we start with the external connections. These external connections are between two adjacent high-level cells. All blue cells have an external connection between themselves and the blue cell they are adjacent to that sits on another high-level yellow cell.
After we’ve found and cached all external connections, we need to handle the internal connections. These are done by just finding a path with HPA* and checking if all points are in a cell.</p>
<p>In the image below we can see the internal connections for the red cell.
Green and blue connections are the same thing, I used a different color to make things a bit more visible.</p>
<p><img src="/images/hierarchical_pathfinding/internal_connections.png" alt="internal connections" /></p>
<h2 id="getting-a-path">Getting a path</h2>
<p>Now that we have our high-level cells and they have connections between them and internal connections we just need to explore two things when we are searching for a path.</p>
<ol>
<li>From the starting point find all possible high-level connections that we can start from</li>
<li>Traverse the high-level graph until we reach the high-level cell containing the destination.</li>
<li>Once we reached that end high-level cell, do a low-level A* to find if from the entry point we have a path to the destination.</li>
</ol>
<p>As our entity moves through the world it will encounter a cell that’s connected to another far away cell. This is for the case where we need to traverse a high-level cell. In that case we have to call A* pathfinding again for that high-level cell.</p>
<p>It is also possible to cache the path and just querry it as it saves us a A* search for a 10x10 cell. This is what I do and if you have spare memory to cache these paths I highly recommend doing so.</p>
<p>For example a path from the start point <code class="language-plaintext highlighter-rouge">S</code> to the destination <code class="language-plaintext highlighter-rouge">D</code> will look like this:</p>
<ul>
<li>The orange cells are part of low-level paths.</li>
<li>The red cells are connected in the high-level pathfinding grid. For them we either have low-level paths cached or we compute them as needed. When we reach the first cell touched by that red arrow.</li>
</ul>
<p><img src="/images/hierarchical_pathfinding/path_found.png" alt="generated path" /></p>
<h2 id="depth--digging">Depth & digging</h2>
<p>Handling depth is simple. For each cell (not only the edges) we check if there is a cell below or above that we are connected to. These cells are kept in case we need to descend to a lower level as we search for a path to our destination.<br />
Handling terrain destruction is simple as we only need to re-compute the high-level connections in the cell where the digging happened.</p>
<h2 id="the-end">The end</h2>
<p>So this is our short journey into HPA* for now. It’s a simple technique to speed up pathfinding for your games, especially if you’re using A* as this comes at an easy integration with your existing pathfinding code. You have to change the core code just slightly and most of the work is done in the initial high-level cell generation. After implementing it the worst gase went from <code class="language-plaintext highlighter-rouge">7ms</code> to <code class="language-plaintext highlighter-rouge">1ms</code>-<code class="language-plaintext highlighter-rouge">1.5ms</code>. There are still ways of improving this, like using the <em>door</em> system that they describe in the paper, but for now I am pleased with the results.
Other ways to speed up pathfinding is using alternative data structures and things like nav meshes.</p>
<p>It’s also important to remember that this can go to more than 1 level of hierarchy above the low-level cells. I didn’t need to generate another level above this one, but I think for certain games it may be useful to keep in mind that you can extend this.</p>
<p>I also explained this on my stream a while back so if you like this post in a video format you can watch that explanation <a href="https://youtu.be/qSbSb8vMbLI?t=915">here</a>.</p>Alexandru EnePathfinding gave me a lot of problems as I went with the clasic A* approach for my game. This worked well except one case. When you don’t have a path to a certain point, you might end up traversing all the blocks of a world desperately trying to find one. This can be quite time consuming, and even for a relatively small 100x100 world, you might spend 7ms searching for a path that doesn’t exist. 7ms is a huge amount of time from a game frame of 16ms so I had to do something about this.After Hours Game Development2019-01-15T00:00:00+00:002019-01-15T00:00:00+00:00http://alexene.dev/2019/01/15/After-hours-game-development<p>In my spare time I am working on a dwarf colony management game that’s written in rust.<br />
I started this project about one year ago and since it has reached this milestone and I didn’t abandon it I think it’s a good time to look at the curent status.</p>
<h2 id="what-is-this-about">What is this about?</h2>
<p><img src="/images/dwarf_game/dwarf_game_january.gif" alt="Dwarves" /></p>
<p>So this is how the game looks on the current build.</p>
<h2 id="general-architecture">General architecture</h2>
<p>The game has an <strong>E</strong>ntity-<strong>C</strong>omponent-<strong>S</strong>ystem (<strong>ECS</strong>) architecture. There are many places where this is explained better so I’m going to be lazy and just link them here:</p>
<ul>
<li><a href="https://www.youtube.com/watch?v=W3aieHjyNvw">Overwatch Gameplay architecture</a></li>
<li>RustConf 2018 closing keynote as <a href="https://www.youtube.com/watch?v=P9u8x13W7UE">video</a> or <a href="https://kyren.github.io/2018/09/14/rustconf-talk.html">text</a>.</li>
</ul>
<p>My quick TLDR explanation is as follows. This architecture is formed out of three parts:</p>
<ul>
<li>Entities - IDs that refer to different components.</li>
<li>Components - Data without behavior.</li>
<li>Systems - Behavior that has no state. Each system acts on a collection of components.</li>
</ul>
<p>I didn’t use <a href="https://github.com/slide-rs/specs">specs</a> or any of the other rust libraries because I started this project as a learning experiment that slowly transformed into a game. If I would start now I would take a serious look at specs before rolling my own implementation. There are some advantages of existing libraries over what I have most of them around boilerplate code.</p>
<p>To create an entity right now I just edit some mega json file called: <code class="language-plaintext highlighter-rouge">recipe_table.json</code>. For example the <code class="language-plaintext highlighter-rouge">Baguette</code> entity recipe looks like this:</p>
<div class="language-json highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="w"> </span><span class="nl">"Baguette"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nl">"can_craft"</span><span class="p">:</span><span class="w"> </span><span class="kc">true</span><span class="p">,</span><span class="w">
</span><span class="nl">"items_list"</span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="w">
</span><span class="err">//List</span><span class="w"> </span><span class="err">of</span><span class="w"> </span><span class="err">items</span><span class="w"> </span><span class="err">needed</span><span class="w"> </span><span class="err">to</span><span class="w"> </span><span class="err">craft</span><span class="w"> </span><span class="err">this</span><span class="w"> </span><span class="err">entity.</span><span class="w">
</span><span class="p">{</span><span class="w">
</span><span class="nl">"entity_type"</span><span class="p">:</span><span class="w"> </span><span class="s2">"Grains"</span><span class="p">,</span><span class="w">
</span><span class="nl">"count"</span><span class="p">:</span><span class="w"> </span><span class="mi">1</span><span class="p">,</span><span class="w">
</span><span class="nl">"processed"</span><span class="p">:</span><span class="w"> </span><span class="kc">false</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">],</span><span class="w">
</span><span class="nl">"components"</span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="w">
</span><span class="err">//Data</span><span class="w"> </span><span class="err">to</span><span class="w"> </span><span class="err">initialize</span><span class="w"> </span><span class="err">various</span><span class="w"> </span><span class="err">components</span><span class="w">
</span><span class="p">{</span><span class="w">
</span><span class="nl">"Item"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="err">//Almost</span><span class="w"> </span><span class="err">all</span><span class="w"> </span><span class="err">entities</span><span class="w"> </span><span class="err">have</span><span class="w"> </span><span class="err">this.</span><span class="w">
</span><span class="nl">"can_carry"</span><span class="p">:</span><span class="w"> </span><span class="kc">true</span><span class="p">,</span><span class="w">
</span><span class="nl">"entity_type"</span><span class="p">:</span><span class="w"> </span><span class="s2">"Baguette"</span><span class="p">,</span><span class="w">
</span><span class="nl">"blocks_pathfinding"</span><span class="p">:</span><span class="w"> </span><span class="kc">false</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">},</span><span class="w">
</span><span class="p">{</span><span class="w">
</span><span class="nl">"Renderable"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="err">//This</span><span class="w"> </span><span class="err">is</span><span class="w"> </span><span class="err">quite</span><span class="w"> </span><span class="err">clear</span><span class="w">
</span><span class="nl">"texture_type"</span><span class="p">:</span><span class="w"> </span><span class="s2">"Baguette"</span><span class="p">,</span><span class="w">
</span><span class="nl">"width"</span><span class="p">:</span><span class="w"> </span><span class="mi">64</span><span class="p">,</span><span class="w">
</span><span class="nl">"height"</span><span class="p">:</span><span class="w"> </span><span class="mi">64</span><span class="p">,</span><span class="w">
</span><span class="nl">"layer"</span><span class="p">:</span><span class="w"> </span><span class="mi">1</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">},</span><span class="w">
</span><span class="p">{</span><span class="w">
</span><span class="nl">"Consumable"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="err">//Consumable</span><span class="w"> </span><span class="err">entities</span><span class="w"> </span><span class="err">can</span><span class="w"> </span><span class="err">be</span><span class="w"> </span><span class="err">used</span><span class="w"> </span><span class="err">by</span><span class="w"> </span><span class="err">dwarves</span><span class="w">
</span><span class="err">//to</span><span class="w"> </span><span class="err">trigger</span><span class="w"> </span><span class="err">various</span><span class="w"> </span><span class="err">effects</span><span class="w">
</span><span class="nl">"modifier"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nl">"effect"</span><span class="p">:</span><span class="w"> </span><span class="s2">"Food"</span><span class="p">,</span><span class="w">
</span><span class="nl">"effect_value"</span><span class="p">:</span><span class="w"> </span><span class="mi">60</span><span class="p">,</span><span class="w">
</span><span class="nl">"effect_type"</span><span class="p">:</span><span class="w"> </span><span class="s2">"Instant"</span><span class="p">,</span><span class="w">
</span><span class="nl">"ticks_since_last_applied"</span><span class="p">:</span><span class="w"> </span><span class="mi">0</span><span class="p">,</span><span class="w">
</span><span class="nl">"should_stack"</span><span class="p">:</span><span class="w"> </span><span class="kc">true</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">],</span><span class="w">
</span><span class="nl">"workbench_type"</span><span class="p">:</span><span class="w"> </span><span class="s2">"CookingTable"</span><span class="w">
</span><span class="p">}</span><span class="w">
</span></code></pre></div></div>
<p>As primitive and ugly it is compared with modern engines UIs, I am really happy with this system. It allows me to quickly create and modify entities. Above all it just works and it allows me to focus on different aspects of the game.</p>
<p>The component properties that you see here aren’t deserialized directly into a component, but a <code class="language-plaintext highlighter-rouge">ComponentDescriptor</code>. A <code class="language-plaintext highlighter-rouge">RenderableComponent</code> has many other members, but the data in his corresponding <code class="language-plaintext highlighter-rouge">RenderableComponentDescriptor</code> is enough to instantiate a <code class="language-plaintext highlighter-rouge">RenderableComponent</code>.</p>
<p>I use <a href="https://github.com/serde-rs/serde">Serde</a> for any serialization and deserialization job and it is an amazing library that is a joy to use and almost invisible. If by any chance you don’t know about it, I recommend you to take a look at some of the example code to get an idea on how it’s used.</p>
<p>I am keeping this as open and configurable as possible in the idea that I want to allow people to mod the game. Potentially someone (not me) can even build some fancy UI instead of working directly with this JSON in the future.</p>
<p>There are some limitations to modding and I will keep them moving forward. New components or systems can’t be added to the game but the existing components can be combined to create new interesting entities (like a plant that attacks dwarves when they pass near it).</p>
<h2 id="how-much-from-scratch">How much from scratch?</h2>
<p>Since people might be interested, here is my <code class="language-plaintext highlighter-rouge">cargo.toml</code> file:</p>
<div class="language-toml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nn">[package]</span>
<span class="py">name</span> <span class="p">=</span> <span class="s">"dwarf_game"</span>
<span class="py">version</span> <span class="p">=</span> <span class="s">"0.1.0"</span>
<span class="py">authors</span> <span class="p">=</span> <span class="p">[</span><span class="s">"Alexandru Ene <alex.ene0x11@gmail.com>"</span><span class="p">]</span>
<span class="py">edition</span> <span class="p">=</span> <span class="s">"2018"</span>
<span class="nn">[dependencies]</span>
<span class="py">sdl2</span> <span class="p">=</span> <span class="s">"0.31.0"</span>
<span class="py">imgui</span> <span class="p">=</span> <span class="s">"0.0.18"</span>
<span class="py">gl</span> <span class="p">=</span> <span class="s">"0.6.0"</span>
<span class="py">memoffset</span> <span class="p">=</span> <span class="s">"0.1"</span>
<span class="py">png</span> <span class="p">=</span> <span class="s">"0.11.0"</span>
<span class="py">serde</span> <span class="p">=</span> <span class="s">"1.0"</span>
<span class="py">serde_derive</span> <span class="p">=</span> <span class="s">"1.0"</span>
<span class="py">serde_json</span> <span class="p">=</span> <span class="s">"1.0"</span>
<span class="py">derive-new</span> <span class="p">=</span> <span class="s">"0.5"</span>
<span class="py">fnv</span> <span class="p">=</span> <span class="s">"1.0.6"</span>
<span class="py">rayon</span> <span class="p">=</span> <span class="s">"1.0"</span>
<span class="py">rand</span> <span class="p">=</span> <span class="s">"0.5.0"</span>
<span class="py">noise</span> <span class="p">=</span> <span class="s">"0.5.1"</span>
<span class="py">lazy_static</span> <span class="p">=</span> <span class="s">"1.2.0"</span>
<span class="py">log</span> <span class="p">=</span> <span class="s">"0.4"</span>
<span class="py">pretty_env_logger</span> <span class="p">=</span> <span class="s">"0.3"</span>
<span class="nn">[dependencies.rust_hawktracer]</span>
<span class="py">version</span> <span class="p">=</span> <span class="s">"0.3.0"</span>
<span class="c">#features=["profiling_enabled"]</span>
<span class="nn">[profile.release]</span>
<span class="py">debug</span> <span class="p">=</span> <span class="kc">true</span>
<span class="nn">[profile.dev]</span>
<span class="py">opt-level</span><span class="p">=</span><span class="mi">0</span>
</code></pre></div></div>
<h2 id="rendering">Rendering</h2>
<p>Rendering is done using OpenGL. I have a small wrapper that provides me with simple abstractions like Texture, ShaderProgram, etc. over <a href="https://github.com/brendanzab/gl-rs">gl-rs</a>. Simplicity is key here and I don’t have anything more than’s strictly need.</p>
<p>I don’t even use texture atlases yet and I just have a bunch of textures dumped into an <code class="language-plaintext highlighter-rouge">assets/images/</code> folder.</p>
<h2 id="ui">UI</h2>
<p>Currently the UI is done using <a href="https://github.com/Gekkio/imgui-rs">imgui-rs</a>. Right now it looks too much like some debug UI so I am unsure if this will be used for the actual game UI as well. I am <em>evaluating</em> the follwing options:</p>
<ul>
<li>Configuring imgui-rs more (Most likely to be the chosen option)</li>
<li>Making my own (I really want to avoid this)</li>
<li>Something else?</li>
</ul>
<p>The best feature of this UI right now is that I can just click on an entity, select it and view all it’s components state so it is an invaluable tool for debugging.</p>
<p>I had to write my own renderer for <code class="language-plaintext highlighter-rouge">gl-rs</code> for <code class="language-plaintext highlighter-rouge">imgui-rs</code>. Nothing complicated, I basically just copied the C++ Opengl example from the main imgui project did and translated it to rust.</p>
<h2 id="terrain">Terrain</h2>
<p>Probably the most invisible big piece of work right now is the terrain.<br />
The terrain is actually 3D and generated from a random seed (think minecraft-like).<br />
Pathfinding works in 3D (so dwarves know when two terrain levels are connected and can go between them), but due to the fact that the current camera is top down and I am such a noob at art, this is impossible to notice unless I explain it.</p>
<h2 id="systems">Systems</h2>
<p>There are a few systems right now at different levels of completion:</p>
<ul>
<li>Combat</li>
<li>Farming</li>
<li>Health</li>
<li>Movement</li>
<li>Pathfinding</li>
<li>Task assignment</li>
<li>Task processing</li>
<li>Needs - hunger and thirst</li>
<li>and few smaller ones</li>
</ul>
<p>Some of them act on a few components, others act on a ton of things. For example the TaskAssignment and TaskProcessing system need to have access to almost all components.</p>
<p>The beauty of this kind of solution is that that complexity and dependency is explicit and contained in one system.
It’s clear from just taking a look at the system data that this is complicated, and not hidden away behind other things.</p>
<p>Systems act on collections of components. For example the combat system has the following view:</p>
<div class="language-rust highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nd">#[derive(new)]</span>
<span class="k">pub</span> <span class="k">struct</span> <span class="n">CombatData</span><span class="o"><</span><span class="nv">'a</span><span class="o">></span> <span class="p">{</span>
<span class="n">dwarf_components</span><span class="p">:</span> <span class="o">&</span><span class="nv">'a</span> <span class="k">mut</span> <span class="n">DwarfComponentContainer</span><span class="p">,</span>
<span class="n">combat_log</span><span class="p">:</span> <span class="o">&</span><span class="nv">'a</span> <span class="k">mut</span> <span class="n">CombatLog</span><span class="p">,</span>
<span class="n">item_components</span><span class="p">:</span> <span class="o">&</span><span class="nv">'a</span> <span class="k">mut</span> <span class="n">ItemComponentContainer</span><span class="p">,</span>
<span class="n">terrain</span><span class="p">:</span> <span class="o">&</span><span class="nv">'a</span> <span class="k">mut</span> <span class="n">Terrain</span><span class="p">,</span>
<span class="n">body_part_components</span><span class="p">:</span> <span class="o">&</span><span class="nv">'a</span> <span class="k">mut</span> <span class="n">BodyPartComponentContainer</span><span class="p">,</span>
<span class="n">armor_components</span><span class="p">:</span> <span class="o">&</span><span class="nv">'a</span> <span class="n">ArmorComponentContainer</span><span class="p">,</span>
<span class="n">transform_components</span><span class="p">:</span> <span class="o">&</span><span class="nv">'a</span> <span class="n">TransformComponentContainer</span><span class="p">,</span>
<span class="p">}</span>
</code></pre></div></div>
<p>It also contains the <code class="language-plaintext highlighter-rouge">update_combat</code> function:</p>
<div class="language-rust highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">pub</span> <span class="k">fn</span> <span class="nf">update_combat</span><span class="p">(</span><span class="n">combat_data</span><span class="p">:</span> <span class="n">CombatData</span><span class="p">)</span> <span class="k">-></span> <span class="nb">Vec</span><span class="o"><</span><span class="n">Action</span><span class="o">></span> <span class="p">{</span>
<span class="o">...</span>
<span class="p">}</span>
</code></pre></div></div>
<p>In <code class="language-plaintext highlighter-rouge">world.update()</code> we just do the following:</p>
<div class="language-rust highlighter-rouge"><div class="highlight"><pre class="highlight"><code> <span class="nn">systems</span><span class="p">::</span><span class="nn">combat</span><span class="p">::</span><span class="nf">update_combat</span><span class="p">(</span><span class="nn">systems</span><span class="p">::</span><span class="nn">combat</span><span class="p">::</span><span class="nn">CombatData</span><span class="p">::</span><span class="nf">new</span><span class="p">(</span>
<span class="o">&</span><span class="k">mut</span> <span class="k">self</span><span class="py">.dwarf_components</span><span class="p">,</span>
<span class="o">&</span><span class="k">mut</span> <span class="k">self</span><span class="py">.combat_log</span><span class="p">,</span>
<span class="o">&</span><span class="k">mut</span> <span class="k">self</span><span class="py">.item_components</span><span class="p">,</span>
<span class="o">&</span><span class="k">mut</span> <span class="k">self</span><span class="py">.terrain</span><span class="p">,</span>
<span class="o">&</span><span class="k">mut</span> <span class="k">self</span><span class="py">.body_part_components</span><span class="p">,</span>
<span class="o">&</span><span class="k">self</span><span class="py">.armor_components</span><span class="p">,</span>
<span class="o">&</span><span class="k">self</span><span class="py">.transform_components</span><span class="p">,</span>
<span class="p">));</span>
</code></pre></div></div>
<p>This makes it clear what the combat system is modifying (terrain, dwarf components, body parts, etc.).
Armor components are read-only since armor doesn’t get damaged in combat yet.</p>
<p>This is way more primitive compared to the way things work in specs or other similar projects where you get an iterator with one dwarf component, one item_component, etc. I have a bit more boilerplate to write in order to get to the same result, but I don’t mind that.</p>
<h2 id="components">Components</h2>
<p>There are a bunch of components such as:</p>
<ul>
<li>Renderable</li>
<li>Armor</li>
<li>Area</li>
<li>Farm</li>
<li>Workbench</li>
<li>Dwarf</li>
<li>Consumable</li>
<li>BodyPart</li>
</ul>
<p>As I said before all components contain data and no functions (except simple getters/setters).</p>
<h2 id="tests">Tests</h2>
<p>Rust makes so easy to add tests that it’s silly not to write them.
I usually tests for bugs I find. Most of them are like integration tests that test how various systems interact. Each time I find a bug I usually try and add a test for it.</p>
<p>For example, this is one bug I had to solve:<br />
When dwarves got hungry as they were crafting something, as they went to eat a baguette they left that task in a limbo state and it couldn’t be finished.</p>
<p>After I fixed it, it was simple to add a test that checks tasks gets handled properly in that case:</p>
<div class="language-rust highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nd">#[test]</span>
<span class="k">fn</span> <span class="nf">test_hungry_dwarf_eats_and_finishes_task</span><span class="p">()</span> <span class="p">{</span>
<span class="k">let</span> <span class="k">mut</span> <span class="n">world</span> <span class="o">=</span> <span class="nn">World</span><span class="p">::</span><span class="nf">new</span><span class="p">();</span>
<span class="n">world</span><span class="nf">.spawn_from_entity_type</span><span class="p">(</span><span class="s">"WorkbenchWoodsmith"</span><span class="p">,</span> <span class="nn">Vec3</span><span class="p">::</span><span class="nf">new</span><span class="p">(</span><span class="mi">0</span><span class="p">,</span> <span class="mi">0</span><span class="p">,</span> <span class="mi">0</span><span class="p">));</span>
<span class="n">world</span><span class="nf">.spawn_from_entity_type</span><span class="p">(</span><span class="s">"WorkbenchAtDestination"</span><span class="p">,</span> <span class="nn">Vec3</span><span class="p">::</span><span class="nf">new</span><span class="p">(</span><span class="mi">0</span><span class="p">,</span> <span class="mi">0</span><span class="p">,</span> <span class="mi">0</span><span class="p">));</span>
<span class="k">let</span> <span class="n">dwarf_id</span> <span class="o">=</span> <span class="n">world</span><span class="nf">.spawn_from_entity_type</span><span class="p">(</span><span class="s">"Dwarf"</span><span class="p">,</span> <span class="nn">Vec3</span><span class="p">::</span><span class="nf">new</span><span class="p">(</span><span class="mi">0</span><span class="p">,</span> <span class="mi">1</span><span class="p">,</span> <span class="mi">0</span><span class="p">));</span>
<span class="n">world</span><span class="nf">.spawn_from_entity_type</span><span class="p">(</span><span class="s">"Baguette"</span><span class="p">,</span> <span class="nn">Vec3</span><span class="p">::</span><span class="nf">new</span><span class="p">(</span><span class="mi">0</span><span class="p">,</span> <span class="mi">5</span><span class="p">,</span> <span class="mi">0</span><span class="p">));</span>
<span class="n">world</span><span class="nf">.spawn_from_entity_type</span><span class="p">(</span><span class="s">"Mattress"</span><span class="p">,</span> <span class="nn">Vec3</span><span class="p">::</span><span class="nf">new</span><span class="p">(</span><span class="mi">4</span><span class="p">,</span> <span class="mi">4</span><span class="p">,</span> <span class="mi">0</span><span class="p">));</span>
<span class="k">for</span> <span class="mi">_</span> <span class="n">in</span> <span class="mi">0</span><span class="o">..</span><span class="mi">5</span> <span class="p">{</span>
<span class="n">world</span><span class="nf">.spawn_from_entity_type</span><span class="p">(</span><span class="s">"Plank"</span><span class="p">,</span> <span class="nn">Vec3</span><span class="p">::</span><span class="nf">new</span><span class="p">(</span><span class="mi">3</span><span class="p">,</span> <span class="mi">3</span><span class="p">,</span> <span class="mi">0</span><span class="p">));</span>
<span class="p">}</span>
<span class="k">let</span> <span class="n">dwarf_component</span> <span class="o">=</span> <span class="nf">find_component</span><span class="p">(</span><span class="n">world</span><span class="nf">.get_dwaf_components</span><span class="p">(),</span> <span class="o">&</span><span class="n">dwarf_id</span><span class="p">)</span><span class="nf">.unwrap</span><span class="p">();</span>
<span class="c">//Make the dwarf hungry</span>
<span class="k">loop</span> <span class="p">{</span>
<span class="n">world</span><span class="nf">.update</span><span class="p">(</span><span class="mi">1</span><span class="p">);</span>
<span class="k">let</span> <span class="n">dwarf_component</span> <span class="o">=</span> <span class="nf">find_component</span><span class="p">(</span><span class="n">world</span><span class="nf">.get_dwaf_components</span><span class="p">(),</span> <span class="o">&</span><span class="n">dwarf_id</span><span class="p">)</span><span class="nf">.unwrap</span><span class="p">();</span>
<span class="k">let</span> <span class="n">stats</span> <span class="o">=</span> <span class="n">dwarf_component</span><span class="nf">.get_stats</span><span class="p">();</span>
<span class="k">if</span> <span class="n">stats</span><span class="py">.hunger</span> <span class="o"><=</span> <span class="n">stats</span><span class="py">.hunger_limit</span> <span class="o">+</span> <span class="mi">1</span> <span class="p">{</span>
<span class="k">break</span><span class="p">;</span>
<span class="p">}</span>
<span class="p">}</span>
<span class="c">//Create a task for him</span>
<span class="k">let</span> <span class="n">task</span> <span class="o">=</span> <span class="nn">PlaceItemTask</span><span class="p">::</span><span class="nf">new</span><span class="p">(</span><span class="n">world</span><span class="nf">.generate_entity_id</span><span class="p">(),</span> <span class="nn">Vec3</span><span class="p">::</span><span class="nf">new</span><span class="p">(</span><span class="mi">6</span><span class="p">,</span> <span class="mi">6</span><span class="p">,</span> <span class="mi">0</span><span class="p">),</span> <span class="s">"Bed"</span><span class="p">);</span>
<span class="n">world</span><span class="nf">.add_task</span><span class="p">(</span><span class="nn">Box</span><span class="p">::</span><span class="nf">new</span><span class="p">(</span><span class="n">task</span><span class="p">));</span>
<span class="k">for</span> <span class="mi">_</span> <span class="n">in</span> <span class="mi">0</span><span class="o">..</span><span class="mi">250</span> <span class="p">{</span>
<span class="n">world</span><span class="nf">.update</span><span class="p">(</span><span class="mi">1</span><span class="p">);</span>
<span class="p">}</span>
<span class="c">//Make sure the task ends and the baguette is eaten.</span>
<span class="k">assert</span><span class="o">!</span><span class="p">(</span><span class="n">world</span><span class="nf">.get_entity_by_type</span><span class="p">(</span><span class="s">"Baguette"</span><span class="p">)</span><span class="nf">.is_none</span><span class="p">());</span>
<span class="k">assert</span><span class="o">!</span><span class="p">(</span><span class="n">world</span><span class="nf">.get_entity_by_type</span><span class="p">(</span><span class="s">"Bed"</span><span class="p">)</span><span class="nf">.is_some</span><span class="p">());</span>
<span class="p">}</span>
</code></pre></div></div>
<p>Almost all tests are instantiating worlds and various entities and scenarios. This means that if a test breaks, I just copy the world initialization code to the main game and I can visualize that test scenario and debug it really easy, pausing the simulaltion, inspecting entities with the debug UI, etc.</p>
<h2 id="one-year">One year</h2>
<p><img src="/images/dwarf_game/first_commit.png" alt="One year" /></p>
<p>I started this one year ago and I commited changes to the project constantly.<br />
How did I manage to stay motivated for this long?</p>
<p>The secret to staying motivated for so long is that there is no secret and I wasn’t motivated and hyped to work on it all the time. It’s ok to take breaks. I’ve had periods where I had a lot of activity and pushed a lot of changes followed by weeks I didn’t write a single line of code. Last summer I did almost no work on it for almost two months for example.</p>
<p><strong>When I’m dealing with with big pieces of work that look scary, the only thing that works 100% for me is to just sit down and start writing.</strong></p>
<p>I mainly design by experimenting. I also have a bit of advantage here since I’ve worked in software and gaming for a while so I kind of know how to avoid the most common pitfals. But really, the most important thing is to just sit down and start writing.</p>
<p>Streaming on twitch also helps, even if I don’t have a regular schedule. It brings some sort of order and mini-deadlines. It also brings down distractions (even if sometimes there’s some chatting going on). I never plan what I stream so I have to sit and work through whatever I said I was going to work on, instead of getting distracted by other things.</p>
<p>Other than that, I have no good answers or advice on this topic.<br />
It’s a first for me too since I usually abandon side projects that take more than 2 months. This one kind of stuck with me.<br />
Maybe it’s also the fact that I really enjoy writing rust?</p>
<h2 id="whats-next">What’s next?</h2>
<p>Things that I want to work on next are:</p>
<ul>
<li>Picking a name for it. Really I’ve been postponing this for way too long.</li>
<li>Digging and manipulating terrain</li>
<li>Refine terrain generation and add natural resources</li>
<li>UI</li>
<li>More systems (weather)</li>
<li>AI director for other factions</li>
<li>Wild animals</li>
<li>Professions & military</li>
<li>Temperature</li>
<li>Water</li>
<li>Adding more items and recipes</li>
<li>Many many other things</li>
</ul>Alexandru EneIn my spare time I am working on a dwarf colony management game that’s written in rust. I started this project about one year ago and since it has reached this milestone and I didn’t abandon it I think it’s a good time to look at the curent status.Rust And Game Development2018-11-15T00:00:00+00:002018-11-15T00:00:00+00:00http://alexene.dev/2018/11/15/Rust-and-game-development<p>Rust is excellent for performance crucial applications that run on multi-processor architectures and these two aspects are also critical for game development. Rust has already seen a bunch of interest from games developers like <a href="https://www.rust-lang.org/pdfs/Rust-Chucklefish-Whitepaper.pdf">Chucklefish</a>, <a href="https://twitter.com/repi/status/1060469377500274689">Embark Studios</a>, <a href="https://twitter.com/andreapessino/status/1021532074153394176?lang=en">Ready at Dawn</a>, etc. - but in order to really excel I’d love to organize some structured efforts to improve the ecosystem and I think it would be great if the 2019 roadmap will include game development.</p>
<h2 id="but-what-is-game-development-anyway">But what is game development anyway?</h2>
<p>Games are made of complex systems where a lot of things usually need to happen in a short amount of time. In game development you have to do your work fast. In games you your work 16 milliseconds fast. If you’re lucky you get about 32 milliseconds.</p>
<p>Even if we ignore rendering, you need to do: physics, animation, updating various gameplay systems, AI, pathfinding and it usually doesn’t stop here. That’s a lot of things that need to happen and that’s why C++ is usually language of choice for game engine development.</p>
<h2 id="the-problem-space">The problem space</h2>
<p>I will break down the problem into two. After doing this step, we have two problems, but trust me they are a bit easier.</p>
<h3 id="1-big-engines">1. Big engines</h3>
<p>These are represented by the big companies that build their own, equally big engine (Ubisoft, DICE, Epic, Unity, Lumberyard, etc.).</p>
<p>There are a few restrictions here, most of them are written in C++ under the hood (even Unity). Most didn’t care about ABI compatibility since it’s all compiled at once so this makes communication to a new Rust module less than ideal. I am not saying we need to solve C++ to Rust bindings, but we need to consider how do we solve fitting Rust systems in an existing engine.</p>
<p>Portability to closed systems. This applies to both categories, but it is really important for this one. While Rust is available on many platforms and architectures, that doesn’t mean it just works on console X or console Y and it’s supported out of the box if you write a <code class="language-plaintext highlighter-rouge">hello world</code> program.
Console game development is a strange, NDA-filled space, unknown to many, and while things are moving in the right direction, there are still problems that need to be solved.</p>
<p>These mammoth engines also care about performance. Sure, you might say consoles are powerful today, but as we said in the intro, you only get 16ms and players expect a lot of things to happen in today’s AAA game worlds.
To achieve this kind of performance a lot of optimization work is put into data structure layouts, SIMD, custom memory allocators, etc. Some of these are already doing quite well in Rust, but others not so well. For example custom memory allocators for the standard containers has a great RFC, but it’s not done yet.</p>
<h3 id="2-small-engines--games">2. Small engines / games</h3>
<p>Here we have the small-medium sized companies that don’t use an off-the-shelf engine, like Chucklefish, Killhouse, and many others (even lone-wolf gamedevs like me, I do a game in my spare time in Rust).</p>
<p>I’ve put <em>engines / games</em> in the title place since we have cases where the engine is a custom-built thing for one game. Yes it still happens even today and that’s fine.</p>
<p>I believe that Rust as a language is ready and has enough maturity and features for this to be possible. As I said in the intro, it’s not only me as a mad, game developer who thinks this, others have jumped on board way before me and announced that they will develop their next game fully or by using rust as much as possible.</p>
<p><strong>Why did mention this category if I think we’re there already for most cases?</strong></p>
<p>Because there are unsolved issues and nuisances here too. One problem that I’ve found is that you kind of have to be an expert to make a game and if you start with the wrong path you get into somewhat frustrating situations. Rust punishes you for being wrong more than other languages, and it does at compile time so you have to do things right in order for them to work.</p>
<p>There’s no <em>it works by the power of luck</em> here and sometimes that feels bad.</p>
<p>People have explained it way better than me here and there are resources available, but it’s something to keep in mind - for more info and solutions see Catherine West’s excellent <a href="https://www.youtube.com/watch?v=aKLntZcp27M">Rustconf 2018 keynote</a></p>
<p>In this space, there are also some Rust game engines but compared to Unity tutorials they have a higher barrier of entry. For example, for the <a href="https://www.amethyst.rs/">Amethyst engine</a>, a simple game of Pong starts out with the following code:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>impl<'a, 'b> SimpleState<'a, 'b> for Pong {
}
</code></pre></div></div>
<p><em>What is that?</em> You might cry, but have no fear, I kind of had the same reaction when I saw that a state needed two lifetime annotations. <code class="language-plaintext highlighter-rouge">'a</code> and <code class="language-plaintext highlighter-rouge">'b</code>. There are good reasons for having them, but for someone who wants to write pong it’s a bit scary. It’s scary for me too and I’ve worked in game development for more than 8 years and I write Rust for more than an year quite intensively.</p>
<p>Do I need that even for pong? I would bet that you can rewrite something like Doorkickers or Stardew Valley or any other 2D game in rust without having to annotate many lifetimes.</p>
<p>Amethyst is shaping to be a nice engine but if all you want to do is a 2D game with simple rules, you could get away with simpler abstractions.</p>
<p>Possibly my point is that there is enough space for more engines to appear and address various targets.</p>
<h2 id="the-solution">The solution</h2>
<p>Now that we’ve seen a bit of the space Rust game development, let’s look at what I think would be a solution.</p>
<p><strong>I propose starting a Game Development focused Working Group.</strong></p>
<p>So we made the working group, now what?</p>
<h2 id="what-should-this-rust-game-development-working-group-do">What should this Rust Game Development Working Group do?</h2>
<p>Besides the usual WG tasks, the role of this working group is to find and tackle systemic problems that game developers face as they write their games in Rust.</p>
<p>Gathering these pain points sorting and distributing them them to the teams that handle different parts of the ecosystem is one role.</p>
<p>Communicating and teaching through tutorials, a status of what the problems encountered are and general info of what’s happening in this space.</p>
<p>This isn’t going to touch a single area. It impacts multiple parts of the ecosystem and we need to identify and collaborate in solving any pain points found.</p>
<p>Now for the practical steps I will split my solution into two parts. – <em>this is an ongoing theme with me splitting things in two parts</em></p>
<h2 id="the-short-term">The short term</h2>
<p>For the short term I’d see a focus on the second category. We are almost there, but there are things still missing or confusing.</p>
<p>We need more resources focused on problems game developers face daily. Some of these I’m sure have been solved a few times already.
For example, if you decided to serialize things with Serde, what’s the best way of serializing / deserializing a <code class="language-plaintext highlighter-rouge">Vec<Box<SomeTrait>></code> object? I’ve personally spent probably 4-5 evenings on this problem. I’m certainly not the brightest tool in the shed, but it would be nice to have a bit more posts and information shared on how you could solve certain things that people usually hit in programming with Rust in this space.</p>
<p>Tooling is another subject. RLS is really good but unfortunately it competes with years of effort put into IDEs like Visual Studio.</p>
<p>I know that Windows is a platform that usually doesn’t get much love, but these days it’s the usual development platform used for games. For example, the rust compiler doesn’t even compile on windows with debug enabled due to linking problems (tries to link too many objects).</p>
<p>C++ / C# patterns don’t directly translate to rust. You can switch from C++ to C# really easy since both accept the same kind of patterns with ease. Rust doesn’t like a lot of these (I’m not going to say bad, but let’s say risky) patterns. Unclear hierarchies of things, shared mutable ownership, etc. These is a space where the rust core team focuses on anyway and provides great solutions and <a href="http://smallcultfollowing.com/babysteps/blog/2018/09/24/office-hours-1-cyclic-services/">advice</a>, but it’s worth mentioning them as a potential problem that game developers will face.</p>
<p>Custom allocator support are a must and almost everyone mentions them so I’d advocate that that needs to have a higher priority and that’s why I include it in this section.</p>
<h2 id="the-long-term">The Long term</h2>
<p>This is a bit more painful to get to, not because of technical reasons but because the world is complex.
Changing things in big organizations or systems is hard but not impossible.</p>
<p>I hope for a future when you can just do <code class="language-plaintext highlighter-rouge">cargo build</code> and get a binary that runs on a game console of your choice. I think it’s a better future for everyone: players will experience less crashes from common avoidable causes and developers are enabled by a modern language.<br />
Much of the hard work has been done and I don’t know of any language features that are needed in order to make this rust game development initiative a success. If there are, we should start discussing them and drafting a RFC.</p>
<p>As a general note, I don’t think what’s in the long term category has to start after we finish the short term category, but I just feel that it may take longer to move and change incredibly big systems with a lot of moving parts.</p>
<h2 id="things-that-we-should-think-about">Things that we should think about</h2>
<p>It has been suggested to have a sort of consensus around Amethyst or another game engine or libraries like specs as the go-to engine/frameworks for game development in Rust.</p>
<p>They are amazing projects and there are advantages in having a one-engine / library focused ecosystem (from the point of discoverability and community), but I think that diversity is important, not only at your workplace and life, but also in the tooling and framework space. Not all games have the same requirements and not all games need engines, so it’s important to be open because it’s quite challenging to create a one-size fits all solution.</p>
<p>That’s not to say that we shouldn’t promote these as options and acknowledge progress, but we should discuss if the focus of this working group should be more towards enabling such frameworks and engines to exist rather than having an agreement on a library/engine.</p>
<h2 id="conclusion">Conclusion</h2>
<p>Game development is a field that’s already full of many unknowns and risks. The ultimate goal of this Game Development Working Group is to take away as many risks as possible by making Rust for game development a viable and I would hope default option.</p>
<p>I am really excited for the core team to announce more structured processes for spinning up working groups in 2019 so that we can move this group forward!</p>Alexandru EneRust is excellent for performance crucial applications that run on multi-processor architectures and these two aspects are also critical for game development. Rust has already seen a bunch of interest from games developers like Chucklefish, Embark Studios, Ready at Dawn, etc. - but in order to really excel I’d love to organize some structured efforts to improve the ecosystem and I think it would be great if the 2019 roadmap will include game development.Learning Rust2018-09-09T00:00:00+00:002018-09-09T00:00:00+00:00http://alexene.dev/2018/09/09/Learning-rust<p>While I didn’t post anything on my blog I have been working on a lot of fun projects in my spare time. Most notably, I have been learning Rust. Part of doing that resulted in a few neat side-projects that I will talk about here in more detail.
This is mostly my journey on learning a new language, I am sure there are other ways, but this worked for me.<br />
I started with the rust <a href="https://doc.rust-lang.org/book/">book</a> and then I continued to learn using a project-based approach.<br />
Below you will see a few of the projects I did along the way with a few words on how they helped me learn various parts of the language.</p>
<h2 id="particles">Particles</h2>
<p>Project: <a href="https://github.com/AlexEne/rust-particles">https://github.com/AlexEne/rust-particles</a></p>
<p>After learning the basics, I have started porting one of my old C++ projects to rust. For me this was a good way to compare the two languages in a real-life situation, where I had to use OpenGL, SDL, and other C libraries.<br />
I don’t fully recommned starting with this kind of project since it might not be the smoothest introduction to a new language but I found it useful.</p>
<p>Most of this project is done using compute shaders, so here I mostly learned how to use other C libraries from Rust.</p>
<p><img src="/images/particles.png" alt="particles" /></p>
<h2 id="chip8-emulator">Chip8 Emulator</h2>
<p>Project: <a href="https://github.com/AlexEne/rust-chip8">https://github.com/AlexEne/rust-chip8</a>.</p>
<p>I found out that doing a chip8 emulator is one of the best ways to start learning a new language.
The reason it works so well is because it only uses a few language constructs: for/while/match. It gets you used to the simple data structures like arrays and maps. You also go in a bit more detail by taking a dependency on another crate (in my case minifb).</p>
<p>It’s a lot of fun since you can find binaries for chip8 games that you can run inside your emulator.</p>
<p><img src="/images/rust-chip8.png" alt="rust-chip8" /></p>
<h2 id="advent-of-code">Advent of code</h2>
<p><a href="https://adventofcode.com/">Advent of code</a> is a programming competition running in at the end of the year, and it consists of daily challenges starting from 1 December to 25 December.<br />
This is can help you learn about data structures, file I/O, string manipulation.
I don’t have the solutions from last year, but I did finish quite a few of the challenges.<br />
It’s really fun when more people are participating and you have a mini-leaderboard where you battle with your friends.</p>
<p><img src="/images/advent-of-code.png" alt="advent-of-code" /></p>
<h2 id="raytracing-in-one-weekend">Raytracing in one weekend</h2>
<p>Project: <a href="https://github.com/AlexEne/raytracing-rs">https://github.com/AlexEne/raytracing-rs</a></p>
<p>This is a really popular topic nowdays, with new tech like RTX popping up. I stumbled upon the <a href="https://www.amazon.co.uk/gp/product/B01B5AODD8">Raytracing in one weekend</a> book and this was another fun project that I did in rust, probably the first one where I used parallelism (using <a href="https://github.com/rayon-rs/rayon">rayon</a>).
You get to learn about multithreading, a bit of ray-sphere intersections, materials and basic raytracing theory.<br />
I highly recommend the following books: <a href="https://www.amazon.co.uk/Ray-Tracing-Next-Week-Minibooks-ebook/dp/B01CO7PQ8C/">Raytracing the next week</a> and <a href="https://www.amazon.co.uk/gp/product/B01DN58P8C/">Raytracing the rest of your life</a>.
Both have a lot of information to get you started and then if you really are into this field, you can move on to <a href="https://www.amazon.co.uk/Physically-Based-Rendering-Theory-Implementation/dp/0123750792">Physically Based Rendering from Theory to Implementation</a></p>
<p>In this project I learned how threads work in rust and how pleasant and comforting is to get thread-related errors at compile time.</p>
<p>The Raytracing in one weekend book is true to its title, and after one weekend you end up with this kind of image:</p>
<p><img src="/images/raytracing-rs.png" alt="raytracing-rs" /></p>
<h2 id="making-a-game-in-rust">Making a game in Rust</h2>
<p>The last one in the list for me is doing a game. I’ve been thinking about this new game idea for a while and I wanted to see how Rust works for games.
This game is currently without a title, but the gameplay is close to gnomoria or similar games.</p>
<p>This is my biggest rust project, at about 6000 lines of code, it includes a super simple <em>engine</em> and ECS architecture. This architecture was mostly inspired by the Overwatch GDC talk that presented how overwatch was built on ECS and how everything worked together.</p>
<p>It uses the following libraries:</p>
<ul>
<li>ImGUI</li>
<li>SDL</li>
<li>OpenGL</li>
<li>Serde</li>
<li>Rayon</li>
<li>and a few others</li>
</ul>
<p>ECS worked amazingly well for me and now I have a bunch of features at this early stage:</p>
<ul>
<li>Terrain generation</li>
<li>Combat</li>
<li>Build tree</li>
<li>Task system</li>
<li>Hunger/Thirst and other effects.</li>
<li>Serialization</li>
<li>Armor and equipment</li>
<li>Pathfinding</li>
</ul>
<p>Working on this I learned about traits, serialization, profiling (more on this below)</p>
<p>They are in various degrees of completion but until now everything works great together.</p>
<p><img src="/images/dwarves_with_helmets_eating_baguettes.gif" alt="dwarves" /></p>
<h2 id="rust-hawktracer">Rust-hawktracer</h2>
<p>Project <a href="https://github.com/AlexEne/rust_hawktracer">https://github.com/AlexEne/rust_hawktracer</a></p>
<p>Hawktracer is a lightweight intrusive profiler developed by my coleague at Amazon that was open-sourced a while ago <a href="https://github.com/amzn/hawktracer">https://github.com/amzn/hawktracer</a>.<br />
While working on my game I had some performance issues that were not easy to diagnose using a sampling profiler and since the original hawktracer project offers a C-API, I have started this crate in order to make rust integration more pleasant.<br />
It has no runtime overhead if you disable profiling since everything is macro-based and if you disable it you don’t get any code generated.</p>
<p>I learned a lot about macros, binding generation, cmake crate. Currently I am still working on this in order to get it at an acceptable quality to be an actual crate.</p>
<p>You can find more info on how to integrate it on the above github page, and I encourage you to send feedback and suggest improvements.</p>
<p><img src="/images/rust-hawktracer.png" alt="rust-hawktracer" /></p>
<h2 id="conclusion">Conclusion</h2>
<p>This was my journey in learning rust, taking a project-based approach.
I will continue writing more detailed posts about the things I am working on (especially the game).</p>Alexandru EneWhile I didn’t post anything on my blog I have been working on a lot of fun projects in my spare time. Most notably, I have been learning Rust. Part of doing that resulted in a few neat side-projects that I will talk about here in more detail. This is mostly my journey on learning a new language, I am sure there are other ways, but this worked for me. I started with the rust book and then I continued to learn using a project-based approach. Below you will see a few of the projects I did along the way with a few words on how they helped me learn various parts of the language.