20180216 - Runtime Edit Reload Continue


This is the RERC workflow in the Low-Dependency Vulkan Engine (LDVE).

Compiler
I use GCC on both Linux and Windows via mingw-w64. This was an important simplification and enables GCC C features.

Everything in Two Files
As a minimalist, I do everything from scratch which enables keeping everything simple and small. The engine is in ldve.c the application, including shader source, is in rom.h. I don't include any system headers, instead I build what I need into the ldve.c source file. There is no library dependences except things like kernel32/etc and then Vulkan which I don't link to. Don't have the "but it works fine on my machine problem".

Three Scripts
Development is done via 2 terminal windows and a single text editor session of notepad2. RERC workflow removes the need for a debugger. Don't have bugs, and can make sure things are working as the code is being brought up. The all.bat builds the standalone binary (one file application, requires no support files and no install). For rapid development, dev.bat runs a loop that builds and runs the base binary in dev mode, then waits for a source file to change. If the app crashes it rebuilds and restarts automatically, factoring out the human work. The hot.bat runs a loop which starts the app then rebuilds the actively changing shader along with a DLL version of the base binary, this loop waits until either of the two source files changes in size. The argument to this script chooses which shader gets recompiled. So CTRL+S saving a source file results in instant reload while app is running.

Three Tools
The scripts use three shell tools of which have source in ldve.c and are built using defines in all.bat. The tool_wait.exe binary waits on source files or output binary to change in size, then exits. Was getting two modification time changes on save file, resulting in two reloads, so had to change. The tool_glsl.exe binary takes a shader number and prepends the associated #version and #define before appending the full rom.h file. Builds the input for glslangValidator.exe. The tool_head.exe binary takes the generated SPIR-V output and converts it into a header file for including into the rom.h header.

Script Details
dev.bat
@set hash=0
:loop
gcc ldve.c -march=amdfam10 -std=gnu11 -Ofast -DAPP_=1 -DBIN_=1 -DCPU_=1 -DDEV_=1 -o rom.exe -s ...
@rom.exe
@tool_wait.exe %hash%
@set hash=%errorlevel%
@goto :loop

hot.bat
@if "%1" neq "" goto ok
@echo Usage: hot.bat shaderNumber
@goto eof
:ok
@set num=%1
@tool_wait.exe 0
@set hash=%errorlevel%
:loop
@tool_glsl.exe %num%
@glslangValidator.exe -V tmp.comp
@tool_head.exe %num%
@rom.exe HOT
@set adr=%errorlevel%
gcc ldve.c -march=amdfam10 -std=gnu11 -Ofast -DAPP_=1 -DHOT_=1 -DCPU_=1 -DDEV_=1 -DRAM_ADR=%adr% -o rom.hot -s -shared ...
@tool_wait.exe %hash%
@set hash=%errorlevel%
@goto :loop
:eof

all.bat
@if "%1" neq "" goto ok
@echo Usage: all.bat lastShaderNumber
@goto eof
:ok
@set last=%1
gcc ldve.c -march=amdfam10 -std=gnu11 -Ofast -DTOOL_GLSL=1 -DBIN_=1 -DCPU_=1 -o tool_glsl.exe -s ...
gcc ldve.c -march=amdfam10 -std=gnu11 -Ofast -DTOOL_HEAD=1 -DBIN_=1 -DCPU_=1 -o tool_head.exe -s ...
gcc ldve.c -march=amdfam10 -std=gnu11 -Ofast -DTOOL_WAIT=1 -DBIN_=1 -DCPU_=1 -o tool_wait.exe -s ...
@set num=99
:loop
@set /a num=%num%+1
tool_glsl.exe %num%
glslangValidator.exe -V tmp.comp
@tool_head.exe %num%
@if %num% neq %last% goto loop
gcc ldve.c -march=amdfam10 -std=gnu11 -Ofast -DAPP_=1 -DBIN_=1 -DCPU_=1 -o rom.exe -s ...
rom.exe
:eof

Re-Attaching
All global data in the engine is in a single structure. Running the base dev binary with HOT as the first argument, @rom.exe HOT, causes it to return the compiled address of this single global structure, which is saved via @set adr=%errorlevel%. Which is then passed into the compile step for the DLL via -DRAM_ADR=%adr%. So the generated DLL can continue inside the existing running application.

Self-Including
In order to facilitate everything, the ldve.c engine includes the rom.h app source file and itself a few times after setting defines. There are 4 internal include passes. DEF_ enables the rom.h file to setup defines before ldve.h gets re-included for the DEF_ pass. TYP_ enables structures to go next. RAM_ enables all global data to easily go into the single structure regardless of where it is placed in the code. ROM_ enables source to go last. Thus decoupling source compile order from ordering source appears in a file.

Hot Reload Source
Source below is going to look alien, but the comments hopefully are clear.
///////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////
///////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////
///////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////
///////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////
//=============================================================================================================================
//
// [ROM] OS ENTRY POINTS AND SETUP FOR HOT RELOAD
//
//-----------------------------------------------------------------------------------------------------------------------------
// Each thread enters the program via 'live = Rom(threadIndex,RAM_ADR)'.
// If 'live==0' then the thread exits.
// All threads need to exit the entry point in order to hot reload the application at run-time.
//-----------------------------------------------------------------------------------------------------------------------------
// RomMain(...) - Entry point for the main thread.
//=============================================================================================================================
#if (ROM_ && BIN_)
 O_ S_ void RomMain(S4 argc,U1R *R_ argv){
  #if DEV_
   // If development binary, check for one argument invocation matching 'HOT', and return RAM base address.
   if((argc==2)&&(argv[1][0]=='H')&&(argv[1][1]=='O')&&(argv[1][2]=='T')&&(argv[1][3]==0))RomDie(U4_(U8_(ramMem)));
  #endif
//-----------------------------------------------------------------------------------------------------------------------------
  NsInit();SigInit();ThrInit();ConInit();KonInit();
//-----------------------------------------------------------------------------------------------------------------------------
  ramV->romArgc=argc;ramV->romArgv=argv;C2 entry=Rom;
//-----------------------------------------------------------------------------------------------------------------------------
  while(1){if(entry(0,RAM_ADR)==0)RomDie(0);
   // In development mode, first run is native binary, 2nd and later runs are HOT reloads via DLL/SO.
   #if DEV_
    // Check for a 'rom.hot' which is larger than zero bytes.
    if(MmfSize(U1R_("rom.hot"))){
     // Reset to no entry condition.
     entry=ramV->romEnt=RomNone;BarW();
     // Bump version.
     ramV->romVer=ramV->romVer+1;BarC();
     // Wait for all threads to get out of the ROM, works up to 255 threads.
     ThrAllOut();
     // Close old ROM.
     if(ramV->rom!=0)LibClose(ramV->rom);
     // Ping-pong between file names to avoid delay on Windows.
     U1R path=(ramR->romVer)&1?U1R_("rom.hot.0"):U1R_("rom.hot.1");
     // Replace temp file with HOT ROM, so HOT ROM can be changed again.
     // Replace can fail on Windows for many milliseconds (maybe locks?), retry until success.
     while(MmfReplace(path,U1R_("rom.hot"),0,0)!=0)ThrSleep(1000);
     // Load new ROM, setup new entry point, if load fails this will crash (as designed).
     entry=ramV->romEnt=C2_(LibSym(ramV->rom=LibOpen(path),U1R_("Rom")));BarC();
     // Before continue wait for extra threads to restart.
     ThrAllIn();}
   #endif
  }}
#endif
///////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////
///////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////
//=============================================================================================================================
// [ROM] OS ENTRY POINTS
//=============================================================================================================================
#if (LNX_ && ROM_ && BIN_)
 // Keep compiler from compiling out.
 O_ S_ void __attribute__((used)) main(S4 argc,U1R *R_ argv){LnxInit();RomMain(argc,argv);}
//-----------------------------------------------------------------------------------------------------------------------------
 __asm__(
  ".text\n"
  ".global _start\n"
  "_start:\n"
  "xor %rbp,%rbp\n"
  "pop %rdi\n"
  "mov %rsp,%rsi\n"
  "andq $-16,%rsp\n"
  "call main\n");
#endif
//-----------------------------------------------------------------------------------------------------------------------------
#if (WIN_ && ROM_ && BIN_)
 O_ void main(S4 argc,U1R *R_ argv){RomMain(argc,argv);}
#endif