Grimoire

[PipeDream]

Warcraft hacking toolkit

Regular users: You probably want to download the Jass NewGen Pack assembled by Vex, which includes all the things you need to rock the WE.

Documentation

Grimoire is a framework for building and running warcraft and world editor hacks. Most of the functionality is implemented in injectable DLLs. Two of these DLLs, war3err and wehack, are useful for every map developer.

Wehack fixes several deficiencies in the world editor. It lets you plug map preprocessors such as JASSHelper in to the saving process. This lets you use PJASS for syntax checking, which eliminates the dreaded syntax error crashes. This will save you a lot of time-no more copying from JASSCraft or picking through the w3xtemp backup. Once your syntax parses and you're ready to play, you can start warcraft with the test map button into opengl and or window mode for faster start up and less hassle. I recommend using JASSHelper and PJASS for preprocessors. Place "mapcompiler.exe" and "pjass.exe" into the main grimoire folder and enable mapcompiler in the compiler menu of the world editor. This is useful even for those of you who only work on terrain, for when you're not working with scripts, you can disable the mapcompiler from the "compiler" WE menu to shave all that time off the saving process. Wehack also includes nolimits functionality-you may need to toggle it off/on though to make it catch.


War3err improves warcraft in a similar fashion. There are several extremely difficult to run down scripting errors that will pass a syntax check. War3err catches them, stops warcraft from crashing if it normally would, and tells you where the problem is, so that you can fix it at your leisure. War3err is compatible with battle.net-you can have it running only on your computer while you play long test games with your friends.
Currently, war3err takes care of these common issues:

• Dividing by zero
• Reading an uninitialized variable
• Running too much code without a TriggerSleepAction()
• Group and location leaks - call DestroyGroup(null) / call RemoveLocation(null) to report top five problem spots
• ExecuteFunc("bad_function")
• Player(bad_number)
• Game pauses in single player when you walk away (add "nopause = on" w/o quotes in war3err.conf in main grimoire directory)

Other JASS related crashes (e.g., any of the other crash happy natives) can be tracked down by enabling the bytecode tracer in war3err.conf, by adding the line "bytecodetrace = on" without the quotes.

The other DLLs have less broadly applicable use or require your users to use grimoire.

• japi.dll - Extend common.j with new natives of your own design. Demonstration implementation includes data structures from the STL and a stopwatch for benchmarking. To benchmark your map, import common.j from the jass\. Then, to benchmark a piece of code:
  
  JASS:

  local integer sw = StopWatchCreate()
  local real settime1 = StopWatchMark(sw)
  local real settime2 = StopWatchMark(sw)
  local real t1
  local real t0 = StopWatchMark(sw)
  // Insert code to be benchmarked here
  set t1 = StopWatchMark(sw)
  call BJDebugMsg("dt: "+R2S(t1-t0-(settime2-settime1)))
  

  
  
• listfile.dll - Create a listfile from every file warcraft or WE accesses inside MPQs.
• loadmpq.dll - Inject an MPQ with high priority into warcraft to override files one can not normally override. Could also be used to pass 4MB limit, I imagine.
• nreplace.dll - Replaces GetLocalizedHotkey() with a disk save/load native. Allows one to create maps with state that work on BNet

Also included is mpqutils which provide some rudimentary manipulations of mpq archives via SFmpq.

War3err for 1.24b attached.

Code:

Grimoire 1.5a 4/25/08
http://www.wc3campaigns.net/showthread.php?t=86652

Contributors:
Bryan "PipeDream" Steinbach - bsteinbach@berkeley.edu
Andy "xttocs" Scott
Mike "PitzerMike" Pitzer
Victor "Vexorian" Soliz
Nikolaj "MindWorX" Mariager


Collection of utilities for using warcraft hacks
see other text files in this directory for descriptions of utilities

Quickstart:
Run startwar3.bat or we.bat to begin

Source available via SVN at:
https://w3grimoire.svn.sourceforge.net/svnroot/w3grimoire

Changelog:
1.5a:
 Append var values to read var lines in bytecode logs
 Added an alternative bytecode logger for long logs
 Renamed "pop" to "poparg" and "setreg" to "pop" in bytecode logs
 startinject exehack command split into startinject1 and 2
  startinject1() uses the new method introduced for 1.21b
  startinject2() uses an older method which may work for you if startinject1() does not.


1.5:
 Reworked loader for 1.21b compatibility (xttocs)
  we DLLs now loaded by auxiliary "weload.dll", which runs we.conf.lua.
  old ongameload.dll used for same purpose for war3

1.4:
 findpath.lua bug fix for warcraft not found in registry
 Cheat("war3err_PauseTracer")/Cheat("war3err_ContinueTracer")
 PitzerMike:
  increase map size to 480x480 directly in the editor
  input custom object IDs when you create an object
  improved international support
  Customize test map settings
  


1.3:
nativepack: new native CallByName.
  To use it:
  call InitNatives()
  call CallByName("somefunction")
  call prototypefunction(string arg1, int arg2)
  This replaces the call to prototypefunction with somefunction dynamically.
  It replaces the next instance of a JASS call, so if you do:
  call foo(bar())
  it will replace the call to bar, not foo.
  the argument types don't have to be the same, but the number should be.
  the replacement is immediate, permanent.
  It scans downward, stopping at the end of the current function.
  It ignores native calls while scanning.
  You need to call InitNatives() once per map run before you use CallByName().

w3jdebug:
 debugger = true in war3err.lua
 Cheat("war3err_Break") to set a breakpoint
 Once warcraft reaches a breakpoint and freezes, start a debugging client
 two debugging clients:  w3jdebug (console) pyw3jdebug.py (GUI)
  pyw3jdebug requires python 2.5, wxPython 2.8, pythoncard 0.8.2
 current commands:
  run
  step
  disas
  getvarval <var>
  funcname
  opnum
  locals
  addbp
  delbp
  listbp
  backtrace
  stack

Cheat("DebugMsg: "+msg) prints to log\\war3erruser.txt

Added backtraces to the usual error messages.  Configure with btonerr and btonerrlen in war3err.lua.
btonerr = false restores the one function behavior.

Cheat("war3err_LocalHTSize") reports size of global hashtable
Cheat("war3err_GlobalHTSize") reports size of local hashtable
Cheat("war3err_DumpLocalHT") prints local hash table to war3err log
Cheat("war3err_DumpGlobalHT") prints global hash table to war3err log

lua upgraded to 5.1.2
wehack.mapmodified() reports if WE thinks the map has been modified
RemoveLocation(-1)/DestroyGroup(-1) report info instead of null, null spouts warning
nopause working again in single player
Removed max op count logging (thread death still detected)
Profiling (LogJASSCalls()/LogVariableUse()) a war3err.lua option

1.2:
Menus for Mike's grimext pack embedded
Saving map on longer necessary for test map
More control code farmed out to lua - add your own tools to the menus easily, see example for jasshelper or mike's tools.
ongameload.dll / ongameload.lua - war3err.dll, japi.dll should be loaded from ongameload.dll via ongameload.lua
split nativepack.dll out of japi.dll
war3err.lua replaces war3err.conf
war3err can be enabled/disabled from grimoire menu

function compilemap() in wehack.lua is executed when map compilation is enabled, before running mapcompiler.exe.  Soon mapcompiler execution will be shuffled into lua so you can do order yourself.
function testmap() in wehack.lua is run when testmap is clicked.


1.1d:
Configuration GUI actually included
wehack.lua added for adding custom menu entries
loadmpq.conf switched with loadmpq.lua - now supports paths with spaces
setregstring(key,value,str) and setregdword(key,value,dword) added to lua files

1.1c:
Configuration GUI - wxLua 2.8
NoLimits absorbed into wehack
common.j / native prototypes synced
DebugPrint fixed, must have been broken since 1.1a
exehack should now play nicely with rootkits like AV programs

1.1b:
war3err.dll:  speed improvement of bytecode logging.  If I missed some possible crash causes it might not log the last few bytecodes.
we.lua:  Updated No-Limits to 1.21 (thanks PitzerMike)

1.1a:
war3err.dll: now logs via popen on logd.exe.  Sample logd.exe writes last 1000 lines to bytecode.txt when warcraft crashes/exits.

1.1:
all:  should give much better error messages now that exception handling works
wehack.dll:  --configure option added for mapcompiler
jAPI.dll:  added DebugPrint() native
war3err.dll:  new conf file war3err.conf
	currently has two options
	nopause = on and bytecodetrace = on
	nopause prevents warcraft from pausing, i.e. when alt tabbing
	helps when you want to walk away and let something run for awhile
	bytecodetrace fills bytecode.txt with each opcode warcraft executes.  This file will grow BIG!
	notes types and names of operations along with the function in which the op resides
	Good for getting a look under the hood into the VM or determining where warcraft stops executing on a crash.



1.0:
all: Improved patch independence & multiple map support
war3err.dll:  
  Uninitialized var use prints name of var in addition to function
  Player(Invalid number) Is replaced with Player(0) and an error message
  rudimentary leak tracing of groups/locations
    call DestroyGroup(null) to report function with most created and not destroyed groups
    call RemoveLocation(null) for same purpose
    reports attempted double frees
  ExecuteFunc("non void func") reports error instead of crashing
    distinguishes between functions that don't exist and those with args
wehack.dll: 
  --nopreprocessor menu option added for mapcompiler

0.7b:
mapcompiler exit codes interpreted correctly
Added option to test map by running startwar3.bat to allow you to use whichever hacks
Forced another case of triggers being un enable able when there are syntax errors in custom script

0.7:
war3err.dll:  prints red error messages in game
wehack.dll: configure hacks from WE menu
Extracts common.j / Blizzard.j from map / warcraft
Runs user supplied mapcompiler in grimoire folder with generated map as argument if option is enabled
Use mpqutils if you want to work with war3map.j
Return 0:  Allow test map
Return other:  Disallow test map 

0.6:
ListFile.dll:  Reports all files opened in .mpq by world editor or warcraft.  can use to build archive listfiles.
Loadmpq opens .conf from working directory at exehack startup

0.5c:
loadmpq finally works with gcc
errors now reported through dialog boxes

0.5b:
Fixed CPU thrashing
Added war3err.dll, which detects:
- Divide by zero
- Uninitialized var use
- Op limit overrun
If one of these occurs during JASS executing, the offending function is noted in war3err.txt

0.5:
DLL injection made more reliable
Warcraft found via registry-  now completely path free.  if you don't have warcraft installed, manually set path
Removed pjass integration
Added source for a JAPI implementation
loadmpq occasionally not working should be fixed
ported to mingw/GCC and nasm.

0.4b:
renamed lots of stuff
added some documentation
added loadmpq.dll for overriding war3patch.mpq

0.4a:
added utilities for adding/deleting mpq files

0.4:
embedded lua
added DLL injection
e.g. can use jAPI.dll from test map button (not included)
pjass 0.99 included

0.3a:
fixed command line argument problem
made a nicer tmap.bat that pops up notepad if there are script errors

0.3:
support for 1.20c - 1.20e
allows one to always enable disabled triggers
removes nag dialog when saving with default map description
get rid of prepend with war3mapImported on imported files
execute tmap.bat instead of war3.exe by default
disable syntax checking


Compiling:
I use mingw 5.0.2 (g++ 3.4.2), nasm 0.98.39 and scons 0.96.94.  New scons are broken for mingw.
You'll need python for scons.
copy nasmw.exe to mingw\bin\
Add mingw\bin\; to the path (Control panel -> System -> Advanced -> Environment Variables -> System Variables -> Path)
Add Python25\ to the path (version number irrelevant)
where python/mingw are the directories you installed each to
Type scons in the grimoire directory to compile


Known bugs:
war3mapImported prefix will return while you open the map without the loader

Includes Lua 5.1 http://www.lua.org/


Thanks to
PitzerMike / Vexorian for original windowwe concept and implementation
Zoxc and his WEHelper for keeping me on my toes with competition

[karukef]

This is a very useful tool indeed.

In case it isn't clear, this tool allows anyone to write their own post-processing tools that do stuff to a map when the Test Map button in the World Editor is pressed.

And syntax checking with PJASS is just 100 times better than the built in syntax checker.

Good job!

[Tim.]

I'll approve this, since I trust PipeDream's work. Mike can give the offical say later.

[BertTheJasser]

<8O stunned.
+Rep

[blu_da_noob]

As long as it hasn't changed since I tested, I can say that it works. And I give it two thumbs up :>

[PipeDream]

Apparently it freezes on Win XP. Uploaded work around executable for now, new version coming soonish.

[Vexorian]

you can use relative paths in link files and avoid using batch files that popup ugly console windows.

Edit: When you try to save a new map with the default text and stuff it shows a message with silly characters instead of just removing the warning wrong addresses or something I guess.

It would be cool if you could catch the moment before adding the script to the map instead of when clicking test map.

Doesn't seem to be extracting common.j and blizzard.j correctly

[PitzerMike]

Excellent work, PipeDream. Now that it can also load mpqs this will definitely be the new standard for WE addins.
*reppie*

The .j files didn't get extracted because the path to war3patch.mpq was wrong in tmap.bat. Put ..\ before war3patch.mpq and it'll work.

Code:

extract ..\War3Patch.mpq scripts\common.j common.j
extract ..\War3Patch.mpq scripts\Blizzard.j Blizzard.j

And I can confirm the popup of funny characters when a map has the default name and description.

[PipeDream]

Thanks for the bug reports/fixes. Popup gibberish was indeed an address typo. I am astounded it didn't cause a crash. To fix the lua files that load the editor, change the address in the dialog stopping entry from 0x0055D6CE to 0x0055D6C3.

Code:

-- Stop some other dialog
patchbyte(proc,0x0055D6C3,0xEB)

[PipeDream]

https://webfiles.berkeley.edu/p_d/tools/deop.rar
I'm still going to update the main distribution eventually but until then here's another DLL that a couple people have requested. This one gives the max number of ops an individual thread reaches and reports any functions that hit the limit and are terminated. Start warcraft by running exehack and look at the two log files for the info.
It's not compatible with JAPI.
BTW maxops is 300,000. Read more in this somewhat outdated thread: http://www.wc3jass.com/viewtopic.php?t=2747

Next two planned features, time permitting:
Automatic profiling
BNet compatible hard drive load/save

Update: This dll should now in theory run on any version of warcraft (this means you vex)

[PipeDream]

Error detection update: This version should detect

• Hitting op limit
• Invalid use of triggersleepaction
• Divide by zero
• Uninitialized variable use

https://webfiles.berkeley.edu/p_d/wc3patches/deop.rar

Source available via svn at
svn://w3jdebug.no-ip.org/wargame

Edit: Seems to work on replays, too.

[Strategy Master]

For the dll injection very rarely every so often the dll injection doesn't appear to work as files in the mpq do not overide. No error logs appear either meaning the injection must have worked fine.
Is there a logical computerised reason why this could happen and we have to just live with it or should i just raise the proirity some more?

[PipeDream]

Yes, that old distribution has a myriad of bugs that might cause it. Hopefully this weekend I'll be able to combine all the scattered patches.
With the exception of supporting wewarlock I want to move away from fiddling with the WE, so try wehelper for that half of mpq loading. For warcraft though I will get a cleaner release out.

[Strategy Master]

Ok cheers will await the update

[PipeDream]

Updated. This is geared towards hacking warcraft, but the DLLs should work on WE too.

Code:

0.5b:
Fixed CPU thrashing
Added war3err.dll, which detects:
- Divide by zero
- Uninitialized var use
- Op limit overrun
If one of these occurs during JASS executing, the offending function is noted in war3err.txt

0.5:
DLL injection made more reliable
Warcraft found via registry-  now completely path free.  if you don't have warcraft installed, manually set path in the lua.
Removed pjass integration
Added source for a JAPI implementation
loadmpq occasionally not working should be fixed
ported to mingw/GCC and nasm.

I have no means of testing the mpq loading. It changed somewhat drastically so let me know if it works!

The error detection DLLs will be integrated into this line soon.