Usually they gave us a binary and a source code of the binary.
Whenever you get a file, issuing file command first to it to know what really file is it.
You need strong in Assembly Language, computer architecture, C programming (Reverse engineering) and Python language to make script for this challenge!
checksec check the properties of executable of binary security.
Stack Canaries = a secret value placed on the stack which changes every time the program is started. the stack canary is checked and if it appears to be modified, the program exits immeadiately.
Nx = stored input or data cannot be executed as code
Address Space Layout Randomization (ASLR) = The randomization of the place in memory where the program, shared libraries, the stack, and the heap are.
RELRO = makes binary sections read-only.
Gdb debugger. Peda, pwndbg or gef.
readelf -a <filename> command. It will displays information about ELF files.
nm <filename> command to know what symbol being called in the binary.
Some tips from expert.
Function that can lead to bof