Yes, that's the point. You need to memorize which words go with la and which with le. Or der/die/das for German. Or no articles for Slavic languages but the declination and other words in the sentence (selection of pronouns, forms of adjectives and sometimes verbs) depend on the gender.
If I remember correctly from my German class in highschool, the rule of thumb was if it's an inanimate object use the feminine Die. That was in the mid 90s and I haven't spoken German since, so that with a grain of salt.
The more accurate rules of thumb are based on word endings. -e or -in suggest it’s feminine, -er or -or that it’s masculine, and -chen or -ling that it’s neutral. Such hints only work for about 30 % of words but some are close to 100% accurate.