Chip investors have had a terrible year. The irony is that things may need to get worse before they get better. 

Micron ‘s rough fiscal first-quarter results late Wednesday added more pressure to the group that is facing its worst annual performance since the financial crisis. The PHLX Semiconductor Index slid more than 4% on Thursday and is now down 36% for the year. That follows three barnstorming years, where the index has averaged a market-trouncing gain of 51% annually. 

But in what is a typical pattern for the cyclical industry, stocks turned down well before the actual numbers did. By the end of June, the SOX was down 35% for the year, while global semiconductor sales for the first six months were showing a 23% gain over the same period in 2021, according to data from the World Semiconductor Trade Statistics trade group. Chip stocks turned down early because experienced investors well understood that the shortages sparked early in the pandemic wouldn’t last, and would in fact drive up inventories that would sooner or later spark a slowdown in sales. 

That is playing out now. Micron saw its own inventory balloon 26% during the most recent quarter to a record high $8.4 billion. The memory-chip maker is no stranger to industry cycles, but that is still the largest sequential jump in inventory Micron has seen in nearly a decade. Chief Executive Officer Sanjay Mehrotra said on Wednesday’s call that the industry is experiencing “the most severe imbalance between supply and demand” for memory in the last 13 years. 

The key question now is: When will chip stocks hit bottom? In his initiation report on the industry last month, Chris Caso of Credit Suisse noted that “historically two-thirds of the pullback in semi stocks occurs before the first bad news hits the tape, and most times they reach a bottom just a few months later.” 

But he also allowed that the current cycle is an unusual one, as certain chip sectors such as analog and automotive have yet to experience a downturn. Many of those buyers were burned by the shortage and have taken to building up larger amounts of inventory to avoid being caught flat-footed again. Four of the largest chip companies serving these markets— Texas Instruments, NXP, Microchip and Analog Devices —averaged 24% year-over-year revenue growth in their most recent quarters. 

Those companies will likely have to start showing weakness for investors to get positive on the broad chip category again. That may still take a while. In a Dec. 14 report, Chris Rolland of Susquehanna noted that while industrywide lead-times, which measure the length of time between chip orders and delivery, have come down over the past few months from record highs, the analog and MCU chip categories—the later a key component in cars—were still two-to-three times their prepandemic levels in November. 

There is some risk for investors in waiting too long to jump back in though. Mr. Caso of Credit Suisse says chip stocks tend to turn up before fundamentals do. He has a positive rating on Micron for that reason, writing that “in a cyclical space such as memory, we’re more inclined to be constructive when suppliers sound this bad” in a note to clients Thursday. Chip investors may need to start stocking up before chip makers clear their shelves.

Write to Dan Gallagher at dan.gallagher@wsj.com