Creating a vignetting model is a very tedious task. The problem is that vignetting depends not only on the focal length, like TCA and distortion. It also depends on aperture and distance to subject. This makes the number of test shots required for a good vignetting model grow exponentially. The TIFFs used must be in the sensor's linear RGB colour space. For example, if you use dcraw, call it like this:
dcraw -4 -o 0 -M myimage.RAW
Hugin uses the following equation for modelling lens vignetting:
Cd = Cs * (1 + k1 * R^2 + k2 * R^4 + k3 * R^6)
(where 'Cd' is destination (resulting) color and Cs is the source color component (does not matter - red, green or blue)).
The good news is that Hugin can automatically compute the k1, k2 and k3 parameters. The bad news is that you'll have to use at least two (I have used three) shots for every focal/aperture/distance triplet and you'll have to align them correctly first (e.g. compute the distortion parameters as described in section Calculating lens distortion. If the images won't be aligned (check this with the "preview" button), the vignetting parameters computed will be absolutely bogus.
Now after you align the images properly, go to the "Exposure" tab and select the following parameters for optimization:
As in the case with distortion, make sure that parameters that we don't need are set to their default values. If this is not the case, switch to the respective tab and rectify that.
Press "Optimize now!" and if all goes well, you'll get the computed k1, k2 and k3 parameters. Pick them up on the "Camera and lens" -> "Photometric" tab and insert into the database. Repeat the above steps for every set of test shots you got. In my case I got something like this:
<lens>
<maker>Pentax</maker>
<model>SMC PENTAX DA 12-24mm F/4 ED AL IF</model>
<mount>Pentax KAF2</mount>
<cropfactor>1.5</cropfactor>
<calibration>
<distortion model="poly3" focal="12" k1="-0.01919" />
<distortion model="poly3" focal="15" k1="-0.00774" />
<distortion model="poly3" focal="18" k1="-0.00345" />
<distortion model="poly3" focal="21" k1="-0.00199" />
<distortion model="poly3" focal="24" k1="0.00061" />
<tca model="poly3" focal="12" br="0.0000916" vr="0.9999904" bb="-0.0002397" vb="1.0006490" />
<tca model="poly3" focal="15" br="0.0001253" vr="0.9999184" bb="-0.0002858" vb="1.0008241" />
<tca model="poly3" focal="18" br="0.0001204" vr="0.9999075" bb="-0.0001990" vb="1.0006258" />
<tca model="poly3" focal="21" br="0.0001063" vr="0.9999562" bb="-0.0001477" vb="1.0005692" />
<tca model="poly3" focal="24" br="0.0000982" vr="1.0000005" bb="-0.0001137" vb="1.0004136" />
<vignetting model="pa" focal="12" aperture="4.5" distance="100" k1="-0.19267" k2="0.09379" k3="-0.38938" />
<vignetting model="pa" focal="15" aperture="4.5" distance="100" k1="-0.08756" k2="-0.28192" k3="0.06908" />
<vignetting model="pa" focal="18" aperture="4.5" distance="100" k1="-0.05982" k2="-0.45748" k3="0.25039" />
<vignetting model="pa" focal="21" aperture="4.5" distance="100" k1="-0.28874" k2="-0.06687" k3="0.09488" />
<vignetting model="pa" focal="24" aperture="4.5" distance="100" k1="-0.44227" k2="0.2599" k3="-0.09436" />
</calibration>
</lens>