Last time we went to Curacao, we followed Losos’ advice and spent some time looking at anole dewlaps.  Matt Brandley wrote up an awesome summary of what we found over at Anole Annals.

What criteria allow you to conclude that X₃ has predictive value when the wAIC for model 1 is only a bit stronger than what I assume is the null model?
The criteria I used here was a difference in AIC score of 3 or more between models. The absolute AIC score is not meaningful but the differences in scores between models can be used as a rough guideline according to Burnham and Anderson in their 2002 book (Model Selection and Multimodel Inference pp 70 and pp 446). They state that “models within 1-2 of the best model have substantial support”. [Although, I have not been able to find a theoretical justification for these rough cut-offs!] Some authors have used a cut-off of 2 and others of 3. I decided to use 3 to be more inclusive of alternative models or more conservative with criteria that a model was ok to stand alone.
When differences (ΔAICc) are within 3 units then that means that those models are plausible too so in order to get an estimate of the effect of the variables included within all plausible models you can do model averaging. Model averaging weights the variables in the model by the AIC weight and adjusts the estimate accordingly. For example, in the model set below, the estimate for “X₃” is adjusted down when you look at the averaged model. However, X₃ still has a positive effect on the response variable Y while X₁ and X₂ have estimates close to zero. Notice also that there is more uncertainty around the estimate for X₃ after averaging. One of the things this suggests to me is that there are relevant X-factors that were not measured and included as potential predictive variables. I have nothing to back this up but the fact that the second-best model is the “null” suggests to me that a lot of the variation might be due to the random factor (not shown in the models below).

Comparing models with small differences in AIC score

After I compared models 1-4 and noticed they were all within 3 ΔAICc of the top model I decided to make model number 5 to see if it would rise to the top. Apparently the penalty of having three times the number of parameters trumped any combined power to predict the response variable.  All of these models also contain site as a random intercept so the variation in y seemed to be due primarily to differences in site and X₃.

What I finally decided to do in this case was to use a cut-off of 2 ΔAICc, not average the models and interpret the results based on the relative weights of the models. …  this suggests that X₃ may have some effect but results are inconclusive.

This is again an example of the general question on what you can say about your results when your wAIC value for your ‘best model’ is not 0.95, but only 0.88 or 0.63, or 0.44.
The weights are calculated using the differences in the AIC score but the weight is also affected by the number of models you have in a set. Since the weights must add to 1, as you add more models some of that weight is “claimed”. So if you have a lot of models in your set the top model can have an AIC weight that is far below say 0.95 but it’s the combined evidence of difference in score, AIC weight and CI is used to determine if competing models are also plausible.
The weights are also useful in model averaging for adjusting the relative contribution of model parameters. Actually, Burnham and Anderson have stressed that model averaging should be done with all hypothesized reasonable models instead of using a cutoff in the score differences. If model averaging is done then parameters that have little impact will have a low estimate (relative to the other parameters in the averaged model if estimates are standardized) and one can see this at a glance. However, standardizing the estimates for different variables may be difficult to impossible. I think their main point is that the primary benefit of model averaging is to develop a more predictive model.

The best model is not necessarily a good model – its just the best out of the ones that you elected to include.
Absolutely correct. However, if you populate the model set with models that can’t predict the response variable then the “null model” (intercept-only model) will be the “best” model. If the null model is the best of the set, this doesn’t necessarily mean “no difference” it means that IF there is a difference it is not explained by the variables in your alternative hypotheses. Additionally, you can use confidence intervals or SE for model estimates in your best models to see how confident you can be that the variables included can explain the variation in the response variable.

How did you choose which models to include?
Personally, I chose to include models that I reasonably thought might be meaningful AND where the independent variable was something I had manipulated in the experiment. Choosing your models is really not different from developing alternative models to test in a study.

For standard stats, if you test 20 hypotheses, you need to adjust your p-value for multiple tests.  Is there an analog with wAICs where when you have so many ‘sets’ that you need to account for the fact that some wAIC of 0.8 or whatever really aren’t meaningful?
When I did the experiment the plan was that I was going to use null-hypothesis testing and that meant I would have to do a long series of “does such-and-such have an effect? yes/no“.
In that case it might make sense to correct for multiple tests as the probability of getting some “yes” answers can increase with the number of times asked. However, (I think) it is more important to use a correction when asking the question multiple times using different explanatory variables (X₁, X₂…) for the same response variable (Y). In my case, I was asking whether stimulus alone affected the outcome and used no other explanatory variables in an attempt to keep “hunting” for a significant result. Therefore, originally I was not incorporating a multiple-test correction in my stats methods.

By using model selection on the surface it does seem that I am asking many times whether variable X₁, X₂, X₃… can predict variable Y. The big difference is that it is not a “game” of probability. If X₁ has any power to explain variation in Y then you get an estimate of the magnitude of that effect with some measure of precision. Rather than just get a yes/no answer that “X₁ has an affect” you get a measure of the magnitude of the effect which is not subject to probability.

You do alter your “chances” of getting models with high AIC weights by using fewer models in a set since each additional model can “claim” some of the total weight of 1 and dilute the pool. However, the weight is not the only selection criteria and there is also not a cut-off so that models below a certain weight are obviously meaningless. That the number of models can affect the spread of weight doesn’t change the chances that your pet alternative hypothesis ends up being the “best” model. If none of the alternative models have predictive power then it’s the “null model” that ends up with the greatest support. This result is not a matter of probability but a matter of developing reasonable, meaningful models.

Personal take:
One of the benefits I’ve seen from using model selection rather than null-hypothesis testing is that it has allowed me to understand what I’m observing in greater detail in a relatively painless way. I know there are more complicated stats (beyond t-tests and non-parametric equivalents) that allow you to see these patterns but I never felt comfortable with the methods. I did not doubt their validity! I simply felt overwhelmed by the assumptions and the need for correcting for multiple tests and basically the logistics of performing a complex MANOVA or worse yet, finding a legitimate non-parametric way to ask the questions I wanted to ask. Fortunately, I had analyzed my data using simple tests and had significant p-values for many of my comparisons. This allowed me to see that model selection was also detecting these differences so the methods were “in agreement”. Model selection allowed me to explore beyond the dichotomous treatment of my data in a way that was more transparent to me. More importantly, it allowed me to get an idea about the impact of different variables rather than a yes/no answer. Again, I’m not saying it can’t be done with “p-value” methods but I have found that for me model selection is more approachable and it helps that there are reasoned arguments to prefer model selection methods over dichotomous p-values assessments. I don’t think for one second (nor have I seen others state) that a shift to model selection as a superior method invalidates experimental results analyzed and evaluated using p-values as criteria.  In my opinion, if anything, perhaps using p-values as criteria rather than model selection has resulted in more type II errors.

http://theoatmeal.com/comics/angler

In the most general sense, here’s how niche modeling works.  First, we have a set of occurrence points for our species:

Each of these occurrences represents an observation of a set of environmental conditions under which we know our species can persist.  Unfortunately, though, this occurrence data almost never comes with direct estimates of all of the environmental factors which are potentially relevant to determining the distribution of the species.  However, we now have very fine-scale data sets containing localized estimates of a bunch of different environmental factors over the entire planet.  We can use that data in conjunction with our occurrence data to estimate the sets of conditions under which our species has been found, by extracting the environmental conditions present at each of those occurrence points.

So where we once just had a bunch of latitude and longitude measurements, we now have a set of conditions that we know our species can handle, at least in the short term (if you look back at the definition of the fundamental niche you’ll see a problem here, but we’ll get to that later).  This is a valuable resource, because we can use these estimates to start building mathematical models of the environmental tolerances of our species.  There are a zillion different methods for building these models, but that’s a discussion for another time.  Suffice to say that we now have a set of points in environment space that we have extracted from our points in geographic space.  Like so:

Environmental niche modeling is simply the application of some algorithm to estimate species tolerances from this sort of data.  They range from very simple heuristics, such as “take the central 95% interval of the species distribution for each environmental variable”, which looks something like this:

To highly complex methods based on algorithms from machine learning that compare the points where the species can be found to the sets of habitats in which they have not been found:

Once we’ve got our model, we can project it back onto the geographic distribution of environmental variables to estimate the suitability of habitat across the entire geographic space.  In summary, the process looks something like this:

Where the colors on the final figure represent the estimated suitability of habitat for our species (note that in this case I just picked a model from the pile I had for California, it is not based on the points in the illustration).

This is a tremendously powerful thing, to the extent that our models can be trusted – it not only tells us something about the environmental tolerances of our species, it can tell us where we might look for populations that we haven’t seen yet.  We can also take the mathematical model of the species tolerances that we constructed in this process and project it onto environmental conditions in other geographic regions to model the ability of the species to invade other areas, or even project it onto estimated conditions in the past or future to estimate the historical or future distributions of species given various scenarios of climate change.

I have glossed over some very serious methodological and conceptual issues with niche modeling in this post, but this is intended as an introduction only.  I’ll get to the ugly bits later.  There are also some serious methodological issues that arise with transferring models to other geographic regions or time periods, but we’ll save those for another day too.  For now we’ll concentrate on the contrast between these methods and the experimental physiological methods I discussed in my last Serious Science Post (i.e., not the post about iguanas farting in the bathtub or various creatures delivering interspecific high fives).

Remember that one of the key issues with physiological niche estimates is their tractability – it is very difficult to get fine-scale estimates of the limits of species tolerances, and becomes increasingly difficult with the addition of more variables.  That’s not anywhere near as much of an issue here – all we have to do to get a new data point for our species is to see it while we have our GPS in hand.  That’s a wonderful thing, because it means that we can amass large amounts of data with minimal effort.  Since environmental variables vary over space, we can also get estimates of the response of our species to a great number of environmental variables at once with the same set of occurrence data.

So what’s the catch?  Well, it actually turns out that there are a heck of a lot of them.  I’ll probably do several posts on key methodological issues with niche models, but I’ll start just by pointing this one out: not all combinations of environmental variables occur in the real world.  Going back to our diagram from before:

We see that our species is perfectly happy in the hottest extremes of our environment space (i.e., the farthest to the right).  What if they were happy in even hotter temperatures?  We have no way of knowing whether that’s true or not, because we’ve already found them in the hottest places available to us!  Likewise, maybe they can handle higher precipitation levels in those hotter climates than they can in colder ones – it certainly looks like there might be a positive correlation between their tolerances for precipitation and temperature.  Once again, we have no way of knowing because there are no extremely hot and extremely wet regions from which we could possibly sample (or fail to sample) our species.  If we were taking an experimental approach we could simply expand the range of conditions we were testing our species under, but that doesn’t work here – as my mother always told me, we can’t afford to heat the whole outdoors.

This is a serious issue – it means that our ability to estimate the niche is limited not only by our number of occurrence points, but by the distribution of environmental variables in the study region.  This issue is serious enough that it has led some people to suggest that we not think of these methods as niche estimates at all, referring to them instead as “species distribution models”.  While I’m sympathetic to that, I still use the term “niche model” for reasons I’ll clarify in a later post.

In summary, niche modeling overcomes a lot of the difficulties that arise with physiological estimation of the niche, but brings a whole slew of other issues along with it.  TANSTAAFL, as always.

Acropora palmata, the elkhorn coral

Corals are unique animals in that they are intimately associated with both their constituent microbial communities and the zooxanthellae that live in the coral tissue. These three components – the coral, the zooxanthellae, and the microbes – make up the coral holobiome. With the advent of next generation sequencing it is possible to characterize the entire community of microbes living on and in association with corals. This allows testing of classic hypotheses related to community assembly. Because the microbial community is intimately associated with the host coral, inferred patterns and processes can “scale up” to the level of the host.

Jacked from gifbin.com.

Update: Matt Brandley has just unearthed additional evidence courtesy of Cute Overload:

In order to understand the effects of climate change on suitability of habitat for a species, we need to know what exactly causes habitat to be more or less suitable for that species.  Does it like high temperatures or low temperatures?  Is the maximum temperature more important than the average temperature?  Does it like wet or dry environments?  Can it tolerate higher temperatures when it’s wet than when it’s cold (i.e., are its tolerances for environmental factors correlated)?  Which environmental factors are important and which are unimportant?

What we really want to know is a thing called the species’ fundamental niche.  This is the range of environments within which that species can maintain a population that reproduces frequently enough that the population does not shrink.  The fundamental niche does not consider interactions between species or the ability of species to disperse to different patches of habitat, it only considers what sets of environmental factors a species can tolerate.  For the sake of illustration, we’re going to talk about the fundamental niche of a cartoon frog:

Here we see an environmental space in two variables: average temperature and annual precipitation.  Within the set of environments circumscribed by the green ellipse, the frog is happy, so happy that it’s reproducing at a sufficient rate to persist.  Outside of the green area the frog is not so happy.  Far enough outside of the green ellipse, the frog is dead.  In order to know whether a particular habitat is suitable for our frog, all we need to know is whether or not it falls inside that green ellipse.  Easy, right?

The answer is “no, not at all”.  It turns out to be a very difficult question.

One possible approach to answering this question is to grab a bunch of our frogs and bring them into the lab.  We can then raise them in a range of temperatures, keeping everything else constant.

We see that our frog does okay when it’s not too hot or too cold, and we can get some idea of what the range of its temperature tolerances is.    Then we can do the same for precipitation:

So what’s the problem with this physiological approach to estimating species tolerances?  Well, it’s actually been a very productive area of research, but there are some issues that limit what we can do with this approach.  One is that the resolution of the experiment is limited by the amount of animals you’re able to raise and your ability to maintain fine differences in the environments they experience.  Despite my little cartoons here, you can’t really just do this with one frog in each box.  You need quite a few so that you can average over all of the statistical noise that’s created by differences between individual frogs and the unavoidable effects of random chance.  In the example above, let’s pretend that each one of those precipitation boxes corresponds to one more foot of rain per year.  Zero and one foot per year are unsuitable, as is anything over seven.  But how about six and a half?  One and a half?  We actually can’t tell where the cutoff is because the resolution is too low!  Assuming we’ve got ten frogs per box, we’ve already used a hundred frogs only to find that we can only estimate their tolerances to the nearest foot.

There’s another problem, though.  Remember our frog’s fundamental niche?

That ellipse has a tilt to it, such that our frogs can handle higher temperatures when there is more precipitation.  By examining one environmental variable at a time, there is no way that we can detect this correlation.  Essentially our study looks like the following figure:

We’ve fixed the precipitation at some value, and then we’ve seen that the frog can handle a range of temperatures given that value of precipitation.  However, we can’t necessarily extrapolate that to the entire range of values that the frog could tolerate.  If we had fixed the precipitation value lower, our frog would have been less able to tolerate high temperatures and more able to tolerate low temperatures than what we observed.  If we had fixed the precipitation value higher, the opposite would have been true!  In order to really understand the species’ tolerance for temperature, we have to examine it over a range of precipitations at the same time.  Something like this:

Notice the big problem?  With even the coarse resolution of our experiments (ten treatments per environmental axis), we’ve now got a hundred experiments to conduct!  If we’re doing ten frogs per experiment, we now have a thousand frogs to take care of.  The addition of more variables compounds the problem – with three variables we’d need a thousand treatments and ten thousand frogs, four variables requires ten thousand treatments and a hundred thousand frogs, etc.  We’re going to run out of frog chow and research assistants fairly quickly at this rate.

So far we’ve been talking about frogs.  Frogs are relatively easy to keep in boxes, but that’s not true of every animal.  Picture the above discussion with elephants substituted for frogs and you quickly see that the enterprise is over before it begins – we simply don’t have enough elephant-sized aquaria to make even a low resolution study practical.  There’s also the fact that these experiments may require more individuals than a natural population can spare – if you want to know the environmental tolerances of a species that is only represented by 500 surviving individuals, no governing body is going to give you license to catch half of the extant population and raise them in the lab, particularly in a set of experiments that may result in some of them dying or at the very least failing to reproduce to their maximum capacity.  Finally, consider that some environmental variables that are relevant to a species may be very difficult to manipulate in a laboratory setting.

I don’t want to be misconstrued here – physiological studies of the niche such as those presented here (albeit in cartoon form) are among the most reliable methods for estimating a species’ environmental tolerances.  The above is simply to point out that there are practical limitations to the resolution and complexity of the niche estimates that can be produced this way.  The traditional infomercial approach at this point would have me saying “THERE’S GOT TO BE A BETTER WAY”, at which point I would expound on the glory that is niche modeling.  However, I do not want to imply that niche modeling is a better approach than physiological studies – in many ways, it is significantly inferior.  However, there are some limitations of physiological studies that niche modeling does not share, and there are times when the lack of those limitations is of great importance (e.g., you can easily build a niche model for elephants using a correlational approach, while a physiological approach would be quite difficult).  Niche modeling does have its own limitations, however, and I’ll talk about some of those as we go on.

In my next post, I’ll talk about what the correlational niche modeling approach is, and how it overcomes some of the issues mentioned here.  In later posts I’ll talk about a whole slew of new methodological issues that the correlational approach raises.